Title: Revision / WebServices
On every note of this deck, unless a note says otherwise:
title: Web Services and Web Data
title: Web Services and Web Data

Front: * Unit 1: Web Services
** What is a web service
Back: a service provided by one machine to other machines on the web

Front: * Unit 1: Web Services
** Why use web services — distributed systems
Back: to build software systems with components (modules) distributed across the web
{:height 429, :width 748}

Front: * Unit 1: Web Services
** Why use web services — reuse
Back: to reuse existing solutions rather than building everything from scratch

Front: * Unit 1: Web Services
** Why use web services — mashups
Back: to enable mashups: applications that combine content from multiple sources into a single new service

Front: * Unit 1: Web Services
** What is a mashup
Back: software that uses content from more than one source to create a single new service; e.g. a flight aggregator combining airlines, hotels, and maps

Front: * Unit 1: Web Services
** SOAP vs REST
*** Machine-to-machine paradigms
Back: -SOAP (legacy)
-REST (dominant today)
-GraphQL (newer)

Q: * Unit 1: Web Services
** SOAP vs REST
*** SOAP originated in [...]; developed by [...]
A: year 2000; Microsoft and IBM

Q: * Unit 1: Web Services
** SOAP vs REST
*** SOAP uses complex [...] message format
A: XML-based

Q: * Unit 1: Web Services
** SOAP vs REST
*** SOAP service discovery uses [...]
A: UDDI — Universal Description, Discovery and Integration

Q: * Unit 1: Web Services
** SOAP vs REST
*** SOAP service description uses [...]
A: WSDL — Web Services Description Language

Q: * Unit 1: Web Services
** SOAP vs REST
*** SOAP is considered [...]; REST has won
A: legacy

Front: * Unit 1: Web Services
** SOAP vs REST
*** What is REST
Back: an architectural style (not an official standard) that uses existing web protocols such as HTTP

Front: * Unit 1: Web Services
** SOAP vs REST
*** SOAP vs REST — winner
Back: REST; majority of services today use REST; major providers phasing out SOAP

Front: * Unit 1: Web Services
** SOAP vs REST
*** When is SOAP still preferred
Back: when higher levels of security are required

Front: * Unit 1: Web Services
** SOAP vs REST
*** SOAP vs REST — comparison
Back: Feature | SOAP | REST
Protocol | XML-based protocol | Architectural style (uses HTTP)
Message format | XML only | Any (JSON, XML, etc.)
Discovery | UDDI | HATEOAS / links
Description | WSDL | OpenAPI / informal
Security | WS-Security (robust) | HTTPS + OAuth
Status | Legacy | Dominant today

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** What is HTTP
Back: HyperText Transfer Protocol; the rules allowing web browsers and web servers to talk to each other

Q: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP sits at the [...], on top of all other network layers
A: application layer

Q: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP uses a [...] model; requests are always initiated by the client
A: client-server

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** What is a web resource
Back: the source of web content; can be a static file or a program generating content on demand

Q: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** Resources are identified by a unique [...]
A: URI (Uniform Resource Identifier)

Q: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** URI vs URL — URI uniquely identifies a resource; URL also specifies the [...]
A: location/access mechanism

Q: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP evolved from [...] to [...], preserving request-response semantics
A: HTTP/0.9; HTTP/3

Q: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Messages
**** HTTP has two message types: [...] and [...]; both share the same basic structure
A: request; response

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Messages
**** Three parts of an HTTP message
Back: -start line
-headers
-optional body (entity-body)

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Messages
**** Request start line format
Back: `method request-URL version`

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Messages
**** Response start line format
Back: `version status-code reason-phrase`

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Messages
**** Status codes — overview
Back: Range | Category | Meaning
1xx | Informational | Request received, continuing
2xx | Success | Request received and accepted
3xx | Redirection | Further action needed
4xx | Client error | Bad request or unauthorised
5xx | Server error | Server failed to fulfil valid request

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Messages
**** Status code — 1xx
Back: Informational

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Messages
**** Status code — 2xx
Back: Success

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Messages
**** Status code — 3xx
Back: Redirection

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Messages
**** Status code — 4xx
Back: Client error

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Messages
**** Status code — 5xx
Back: Server error

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Headers
**** Header syntax
Back: `name: value`

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Headers
**** General headers
Back: used by both clients and servers; e.g. `Date:`

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Headers
**** Request headers
Back: provide extra info to the server; e.g. `Accept: */*`

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Headers
**** Response headers
Back: provide info to the client; e.g. `Server: Apache/2.0`

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Headers
**** Entity headers
Back: describe the entity body; e.g. `Content-Type: text/html; charset=utf-8`

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** MIME Types
**** What is MIME
Back: Multipurpose Internet Mail Extensions; originally for email, adopted by HTTP to describe multimedia content

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** MIME Types
**** MIME type structure
Back: `primary-type/subtype`

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** MIME Types
**** MIME type for HTML
Back: text/html

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** MIME Types
**** MIME type for plain text
Back: text/plain

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** MIME Types
**** MIME type for JPEG
Back: image/jpeg

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** MIME Types
**** MIME type for JSON
Back: application/json

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** MIME Types
**** MIME type for XML
Back: application/xml

Q: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** MIME Types
**** MIME types are registered with [...] (Internet Assigned Numbers Authority)
A: IANA

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Methods
**** GET
Back: fetches a resource from the server; data sent in query string, not body

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Methods
**** HEAD
Back: like GET but server returns headers only, no body; used to check resource type, existence, or last-modified date

Q: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Methods
**** PUT [...] a resource to the server at a specific URI
A: writes

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Methods
**** POST
Back: sends data to the server for processing; PUT stores at URI, POST processes

Q: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Methods
**** TRACE — server [...]; lets client see how the request arrived
A: echoes the request back

Q: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Methods
**** OPTIONS — asks server which [...] for a resource
A: HTTP methods it supports

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Methods
**** DELETE
Back: asks server to delete a resource; no guarantee the delete is carried out

Q: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Methods
**** HTTP extension methods — field-extensible so new features don't break older software; e.g. [...]
A: WebDAV

Front: * Unit 1: Web Services
** HTTP — the Workhorse of the Web
*** HTTP Methods
**** HTTP methods — summary
Back: Method | Action | Has body? | Idempotent?
GET | Fetch resource | No | Yes
HEAD | Fetch headers only | No | Yes
PUT | Write/replace resource | Yes | Yes
POST | Process data | Yes | No
DELETE | Delete resource | No | Yes
OPTIONS | List supported methods | No | Yes
TRACE | Echo request back | No | Yes

Q: * REST Constraints
** REST coined by [...], circa 2000, in his [...] on network-based software architectures
A: Roy Fielding; PhD thesis

Front: * REST Constraints
** "Representational" means
Back: how data is represented when sent over the web; representation does not have to match the storage format

Front: * REST Constraints
** "State" means
Back: the current knowledge/condition of a client or server; changes as the client interacts with the server

Front: * REST Constraints
** "Transfer" means
Back: the transfer of a representation from server to client (or vice versa)

Front: * REST Constraints
** REST overall meaning
Back: the change in state of an entity driven by a representation transferred from another entity

Front: * REST Constraints
** Mandatory REST constraints
Back: 5 mandatory; a 6th (Code on Demand) is optional

Front: * REST Constraints
** 1. Client-Server
*** Constraint 1 — Client-Server
Back: client and server are separate; requests are always initiated by the client

Front: * REST Constraints
** 1. Client-Server
*** Client-server advantage — portability
Back: improves portability of the user interface across multiple platforms

Front: * REST Constraints
** 1. Client-Server
*** Client-server advantage — evolution
Back: allows client and server components to evolve independently

Q: * REST Constraints
** 2. Statelessness
*** Statelessness — each request contains [...]; no reliance on previous requests
A: all information necessary to understand it

Front: * REST Constraints
** 2. Statelessness
*** Statelessness advantage — reliability
Back: can recover from partial failure; a broken sequence can continue later

Front: * REST Constraints
** 2. Statelessness
*** Statelessness advantage — visibility
Back: a monitoring system only needs to look at a single request to understand it

Front: * REST Constraints
** 2. Statelessness
*** Statelessness advantage — simplicity
Back: server does not need to store previous request state

Front: * REST Constraints
** 2. Statelessness
*** Statelessness disadvantage — performance
Back: repetitive data must be sent in every request

Front: * REST Constraints
** 2. Statelessness
*** Statelessness disadvantage — control
Back: placing state on the client reduces the server's control over application behaviour

Q: * REST Constraints
** 3. Cacheability
*** Cacheability — responses must be labelled [...]; cacheable responses can be reused for equivalent later requests
A: cacheable or non-cacheable

Front: * REST Constraints
** 3. Cacheability
*** Cacheability advantage
Back: eliminates some interactions, improving performance and reducing latency

Front: * REST Constraints
** 3. Cacheability
*** Cacheability disadvantage
Back: stale cached data may differ from the server, reducing reliability

Front: * REST Constraints
** 4. Uniform Interface
*** Constraint 4 — Uniform Interface
Back: the most important REST constraint; divided into 4 sub-constraints

Q: * REST Constraints
** 4. Uniform Interface
*** 4.1 Identification of Resources
**** Resources are identified by a unique and stable [...]
A: URI

Q: * REST Constraints
** 4. Uniform Interface
*** 4.1 Identification of Resources
**** Unique URI — each identifier points to [...] resource
A: one and only one

Q: * REST Constraints
** 4. Uniform Interface
*** 4.1 Identification of Resources
**** Stable URI — the identifier [...] over time
A: must not change

Front: * REST Constraints
** 4. Uniform Interface
*** 4.1 Identification of Resources
**** Good URI — example
Back: `/users/123`, `/blog/2025/02/03/rest-tutorial`

Front: * REST Constraints
** 4. Uniform Interface
*** 4.1 Identification of Resources
**** Bad URI — verb in path
Back: `/getUser` (verbs break identification)

Front: * REST Constraints
** 4. Uniform Interface
*** 4.1 Identification of Resources
**** Bad URI — not unique
Back: `/users/current` (changes per session)

Q: * REST Constraints
** 4. Uniform Interface
*** 4.2 Manipulation Through Representations
**** Manipulation through representations — client works with a [...] of a resource; server uses it to update the real resource
A: representation

Q: * REST Constraints
** 4. Uniform Interface
*** 4.3 Self-Descriptive Messages
**** Self-descriptive messages must carry [...] to process them; e.g. `Content-Type` must be included
A: all the information the receiver needs

Front: * REST Constraints
** 4. Uniform Interface
*** 4.4 HATEOAS
**** HATEOAS stands for
Back: Hypermedia As The Engine Of Application State

Q: * REST Constraints
** 4. Uniform Interface
*** 4.4 HATEOAS
**** HATEOAS rule — server representations must include [...] to further actions or related resources
A: hypermedia links

Front: * REST Constraints
** 4. Uniform Interface
*** 4.4 HATEOAS
**** HATEOAS is also called
Back: the discoverability rule

Front: * REST Constraints
** 4. Uniform Interface
*** 4.4 HATEOAS
**** HATEOAS example
Back: a shopping cart response includes links for checkout, payment, and other related actions

Q: * REST Constraints
** 5. Layered System
*** Layered System — client has no knowledge of intermediary layers; each layer only aware of the [...]
A: next layer

Front: * REST Constraints
** 5. Layered System
*** Layered system advantage
Back: boosts scalability; separates concerns (security, storage, auth) without burdening the client

Front: * REST Constraints
** 5. Layered System
*** Layered system disadvantage
Back: adds latency because each request must traverse multiple layers

Front: * Bridging the Semantic Gap
** The REST Cycle
*** REST cycle steps
Back: -client requests a representation of a resource
-server sends representation in some format (e.g. HTML)
-client parses the representation
-client changes its state using information in the representation
-client recognises control structures (e.g. HTML links) leading to further resources
-client decides on a new resource to request; repeat

Front: * Bridging the Semantic Gap
** The REST Cycle
*** Step that cannot be fully automated
Back: step 6 — deciding which resource to request next; requires semantic understanding

Q: * Bridging the Semantic Gap
** The Semantic Gap
*** Semantic gap — gap between a document's [...] (parseable by computer) and its [...] (requires semantic programming)
A: structure; meaning

Front: * Bridging the Semantic Gap
** The Semantic Gap
*** Bridge gap — API designer step 1
Back: write application semantics in a human-readable specification

Front: * Bridging the Semantic Gap
** The Semantic Gap
*** Bridge gap — API designer step 2
Back: register IANA media type(s) linking to the specification

Front: * Bridging the Semantic Gap
** The Semantic Gap
*** Bridge gap — client developer step 1
Back: look up the unknown media type in IANA

Front: * Bridging the Semantic Gap
** The Semantic Gap
*** Bridge gap — client developer step 2
Back: read the spec and program the semantics into the client

Front: * Bridging the Semantic Gap
** The Semantic Gap
*** Can the semantic gap be eliminated
Back: no; computers are not as smart as humans; it can only be narrowed

Front: * Bridging the Semantic Gap
** Fiat Standards
*** What is a fiat standard
Back: a description of the way somebody does things; no one agreed to it as a standard; every current API (Twitter, Facebook, Google) is essentially a fiat standard

Front: * Bridging the Semantic Gap
** Link Relations
*** What is a link relation
Back: a string on a hypermedia control that explains what state change will occur if the client triggers it

Front: * Bridging the Semantic Gap
** Link Relations
*** Link relations managed by
Back: IANA; ~60 registered general link relations

Front: * Bridging the Semantic Gap
** Link Relations
*** Registered link relation
Back: a short string in the IANA registry; e.g. `next`, `previous`

Front: * Bridging the Semantic Gap
** Link Relations
*** Extension link relation
Back: URL-shaped; defined by the owner of the domain for a custom meaning

Front: * Bridging the Semantic Gap
** Link Relations
*** RFC 5988
Back: defines the two kinds of link relations: registered and extension

Front: * Bridging the Semantic Gap
** Maze+XML — Semantic Gap Worked Example
*** Maze+XML media type
Back: `application/vnd.amundsen.maze+xml`

Front: * Bridging the Semantic Gap
** Maze+XML — Semantic Gap Worked Example
*** Structured syntax suffix
Back: augmentation that specifies the underlying syntax; `+xml` means XML syntax with custom semantics

Front: * Bridging the Semantic Gap
** Maze+XML — Semantic Gap Worked Example
*** Maze+XML link relations
Back: -maze, start
-east, west, north, south
-exit

Front: * Bridging the Semantic Gap
** Maze+XML — Semantic Gap Worked Example
*** How Maze+XML bridges the semantic gap
Back: defines link relations with specific programmed meanings; client knows what east, west, exit etc. mean because the developer read the spec

Front: * Bridging the Semantic Gap
** Maze+XML — Semantic Gap Worked Example
*** What a Maze+XML client needs to start
Back: only the root URL and knowledge of the Maze+XML spec; all navigation follows from link relations

Front: * Bridging the Semantic Gap
** Media Types
*** text/plain
Back: simplest format; no structure beyond natural language rules; for short human-readable responses

Front: * Bridging the Semantic Gap
** Media Types
*** What is HTML
Back: HyperText Markup Language; oldest and most widely used hypermedia format (1993); uses tags to denote structure; now a living standard

Front: * Bridging the Semantic Gap
** Media Types
*** HTML advantage — support
Back: widely supported; tools and libraries available on all platforms

Front: * Bridging the Semantic Gap
** Media Types
*** HTML advantage — human content
Back: lends itself to human-readable applications with rich features

Front: * Bridging the Semantic Gap
** Media Types
*** HTML disadvantage — machine use
Back: designed for human documents; poor choice for machine-to-machine data

Front: * Bridging the Semantic Gap
** Media Types
*** HTML disadvantage — bandwidth
Back: large amounts of metadata overhead increase bandwidth

Front: * Bridging the Semantic Gap
** Media Types
*** What is XML
Back: Extensible Markup Language (1998); represents pure data regardless of rendering; tags are user-defined with no predefined semantics; `application/xml`

Front: * Bridging the Semantic Gap
** Media Types
*** XML advantage — flexibility
Back: can underlie any media type (e.g. SOAP, RSS, Maze+XML)

Front: * Bridging the Semantic Gap
** Media Types
*** XML disadvantage — weight
Back: ~7 lexical tokens per tag; heavier than JSON

Front: * Bridging the Semantic Gap
** Media Types
*** XML disadvantage — semantics
Back: no built-in semantics; must be defined separately

Front: * Bridging the Semantic Gap
** Media Types
*** RSS
Back: Really Simple Syndication; machine-readable website updates using XML; `application/rss+xml`

Front: * Bridging the Semantic Gap
** Media Types
*** Atom
Back: alternative to RSS; `application/atom+xml`

Front: * Bridging the Semantic Gap
** Media Types
*** SOAP media type
Back: `application/soap+xml`

Front: * Bridging the Semantic Gap
** Media Types
*** What is JSON
Back: JavaScript Object Notation; human-readable, language-independent data format (circa 2001); `application/json`

Q: * Bridging the Semantic Gap
** Media Types
*** JSON is built on [...] (objects/dicts) and [...] (arrays)
A: name-value pairs; ordered lists

Front: * Bridging the Semantic Gap
** Media Types
*** JSON advantage
Back: lower overhead than XML; easy to parse

Front: * Bridging the Semantic Gap
** Media Types
*** JSON disadvantage — comments
Back: does not support comments

Front: * Bridging the Semantic Gap
** Media Types
*** JSON disadvantage — semantics
Back: no inherent semantics; must be defined per use

Front: * Bridging the Semantic Gap
** Media Types
*** YAML
Back: indentation-based (spaces only); key: value pairs; lists with `-`; comments with `#`; supports strings, numbers, booleans, null

Front: * Bridging the Semantic Gap
** RESTful API Design Exercise
*** Rating system — module constraint
Back: a module may be taught by different professors in different years/semesters, or by multiple professors simultaneously

Front: * Bridging the Semantic Gap
** RESTful API Design Exercise
*** Rating system — authentication
Back: users must register (username, email, password) and be logged in to rate

Front: * Bridging the Semantic Gap
** RESTful API Design Exercise
*** Rating system — overall rating formula
Back: average of all ratings for that professor across all module instances; rounded to nearest integer

Front: * Bridging the Semantic Gap
** RESTful API Design Exercise
*** Rating system — client operations
Back: -view all module instances and the professor(s) teaching each
-view the rating of all professors
-view average rating of a professor in a specific module
-rate the teaching of a professor in a specific module instance

Front: * Unit 2: Search Engines
** Search Engine Pipeline
*** Three processes of a search engine
Back: -web crawling
-indexing
-searching

Front: * Unit 2: Search Engines
** Search Engine Pipeline
*** Web crawling
Back: automatically finding and downloading web pages by following hyperlinks

Front: * Unit 2: Search Engines
** Search Engine Pipeline
*** Indexing
Back: associating words found on web pages to their URLs and HTML-based fields

Front: * Unit 2: Search Engines
** Search Engine Pipeline
*** Searching
Back: at query time, retrieve URLs containing keywords from the index and rank them

Front: * Unit 2: Search Engines
** The Deep Web
*** What is the deep web
Back: the part of the WWW invisible to search engines; hidden behind forms, login walls, paywalls, or requiring a direct URL

Front: * Unit 2: Search Engines
** The Deep Web
*** Surface vs deep web
Back: ~5 billion visible pages; deep web is several orders of magnitude larger (iceberg analogy)

Front: * Unit 2: Search Engines
** The Deep Web
*** Condition for a page to be visible
Back: there must be a valid link to it from another publicly accessible page

Q: * Unit 2: Search Engines
** Web Crawling
*** Web crawling is [...] using [...] traversal
A: directed graph traversal; breadth-first

Front: * Unit 2: Search Engines
** Web Crawling
*** The crawling process
Back: -start with seed URLs; add to the URL queue (frontier)
-take a URL from the queue; download the page
-parse for link tags; find new URLs
-add unseen URLs to the frontier
-repeat indefinitely (or until disk runs out)

Front: * Unit 2: Search Engines
** Web Crawling
*** Frontier
Back: the set of all URLs in the queue at any given time

Front: * Unit 2: Search Engines
** Web Crawling
*** Crawling challenge — scale
Back: billions of pages on the web

Front: * Unit 2: Search Engines
** Web Crawling
*** Crawling challenge — dynamism
Back: pages are constantly created, removed, and modified

Front: * Unit 2: Search Engines
** Web Crawling
*** Crawling challenge — control
Back: web pages are not under the search engine's control

Front: * Unit 2: Search Engines
** Web Crawling
*** Crawling challenge — deep web
Back: some data is only accessible by submitting forms

Front: * Unit 2: Search Engines
** Web Crawling
*** Politeness Policies
**** Why politeness policies
Back: fetching many pages at once overloads web servers; admins may block the crawler's IP

Front: * Unit 2: Search Engines
** Web Crawling
*** Politeness Policies
**** Politeness rule 1
Back: fetch no more than one page at a time from any particular web server

Front: * Unit 2: Search Engines
** Web Crawling
*** Politeness Policies
**** Politeness rule 2
Back: wait at least a few seconds (sometimes minutes) between requests to the same server

Front: * Unit 2: Search Engines
** Web Crawling
*** Politeness Policies
**** Politeness rule 3
Back: split the queue into per-server queues; only read from a queue after its politeness window has passed

Q: * Unit 2: Search Engines
** Web Crawling
*** Politeness Policies
**** Politeness window example — crawling 100 pages/sec with a 30s window needs URLs from [...]
A: 3,000+ different servers

Front: * Unit 2: Search Engines
** Web Crawling
*** robots.txt
**** What is robots.txt
Back: a file at `/robots.txt`; specifies which resources crawlers are allowed to access

Front: * Unit 2: Search Engines
** Web Crawling
*** robots.txt
**** robots.txt — User-agent line
Back: identifies which crawler (or group of crawlers) the following rules apply to

Front: * Unit 2: Search Engines
** Web Crawling
*** robots.txt
**** robots.txt — rules
Back: Allow and Disallow lines; optional Crawl-delay (non-standard)

Front: * Unit 2: Search Engines
** Web Crawling
*** robots.txt
**** robots.txt — sitemap
Back: can include a reference to a sitemap

Front: * Unit 2: Search Engines
** Web Crawling
*** robots.txt
**** User-Agent HTTP header
Back: used by crawlers/browsers to declare identity; e.g. `User-Agent: Googlebot/2.1`; can be spoofed

Front: * Unit 2: Search Engines
** Web Crawling
*** Detecting Robots
**** Detection method — quick requests
Back: very fast page requests to the same server are the most obvious crawler sign

Front: * Unit 2: Search Engines
** Web Crawling
*** Detecting Robots
**** Detection method — trap links
Back: hidden links humans won't click; a naive crawler steps in and is detected; avoided by reading robots.txt

Front: * Unit 2: Search Engines
** Web Crawling
*** Detecting Robots
**** Detection method — fictitious resources
Back: dynamically generated infinite link chains; causes naive robots to recurse infinitely; avoided by capping crawl depth

Front: * Unit 2: Search Engines
** Web Crawling
*** Detecting Robots
**** Detection method — User-Agent blocking
Back: blacklisting known bot User-Agent strings; avoided by spoofing

Front: * Unit 2: Search Engines
** Web Crawling
*** Detecting Robots
**** Detection method — monotonous timing
Back: requests at regular intervals are detectable; avoided by randomising timing

Front: * Unit 2: Search Engines
** Web Crawling
*** Detecting Robots
**** Signs a crawler is being blocked
Back: CAPTCHA pages, unusual delay, repeated status codes: 301, 401, 403, 404, 408, 429, 503

Front: * Unit 2: Search Engines
** Web Crawling
*** Page Freshness
**** Why maintain freshness
Back: pages are constantly added, deleted, or modified; index must stay accurate

Front: * Unit 2: Search Engines
** Web Crawling
*** Page Freshness
**** How to check freshness
Back: send HTTP HEAD request (headers only); compare `Last-Modified` with date from previous GET

Front: * Unit 2: Search Engines
** Web Crawling
*** Page Freshness
**** HEAD vs GET for freshness
Back: HEAD avoids downloading the full page but cannot eliminate the cost entirely

Front: * Unit 2: Search Engines
** Web Crawling
*** Page Freshness
**** Adaptive freshness scheduling
Back: a crawler can learn how frequently each page changes and predict when it is next likely to need re-fetching; avoids wasting HEAD requests on stable pages

Front: * Unit 2: Search Engines
** Web Crawling
*** Sitemaps
**** What is a sitemap
Back: an XML file at `/sitemap.xml` listing URLs and metadata

Front: * Unit 2: Search Engines
** Web Crawling
*** Sitemaps
**** Sitemap — ``
Back: the URL of the resource

Front: * Unit 2: Search Engines
** Web Crawling
*** Sitemaps
**** Sitemap — ``
Back: how often the resource changes (e.g. daily, weekly)

Front: * Unit 2: Search Engines
** Web Crawling
*** Sitemaps
**** Sitemap — ``
Back: date the resource was last modified

Front: * Unit 2: Search Engines
** Web Crawling
*** Sitemaps
**** Sitemap — ``
Back: importance relative to other pages on the site (default 0.5)

Front: * Unit 2: Search Engines
** Web Crawling
*** Sitemaps
**** Why sitemaps
Back: tells search engines about pages they might not discover (e.g. those behind forms)

Front: * Unit 2: Search Engines
** Web Crawling
*** Distributed Crawling and Conversion
**** Why distributed crawling
Back: crawling the whole web needs many computers to reduce queue size, get closer to sites, and spread CPU and bandwidth load

Front: * Unit 2: Search Engines
** Web Crawling
*** Distributed Crawling and Conversion
**** Conversion problem
Back: text exists in hundreds of incompatible formats; search engines must convert all of them

Front: * Unit 2: Search Engines
** Web Crawling
*** Distributed Crawling and Conversion
**** Why not convert to plain text
Back: strips headings and font sizes; structural cues (bold, headings) are valuable for ranking

Front: * Unit 2: Search Engines
** Web Crawling
*** Distributed Crawling and Conversion
**** Conversion solution
Back: convert to a tagged format (HTML or XML) to preserve structural information

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** Why links help rank pages
Back: links are votes of importance; the link structure reveals which pages are popular

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** Anchor text
Back: the visible, clickable text of a hyperlink; a short topical description of the linked page

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** Anchor text analysis
Back: for each link, if anchor text matches a query, add 1 to the destination page's score

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** Anchor text analysis — problem 1
Back: doesn't handle ambiguous queries (e.g. "click here")

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** Anchor text analysis — problem 2
Back: doesn't account for the rank of the source page

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** Better use of anchor text
Back: treat anchor text as an additional text field for the destination page

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** Simple link analysis
Back: rank pages by counting inbound links; more links = higher rank

Q: * Unit 2: Search Engines
** Link Analysis and PageRank
*** Simple link analysis — limitation — does not account for [...]
A: source page importance or rank

Q: * Unit 2: Search Engines
** Link Analysis and PageRank
*** PageRank
**** PageRank model is based on the [...] (Alice randomly follows links forever)
A: random web surfer

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** PageRank
**** Why popular pages rank higher
Back: popular pages have more inbound links so the surfer visits them more often

Q: * Unit 2: Search Engines
** Link Analysis and PageRank
*** PageRank
**** PageRank value is the [...] that the random surfer is at that page at any moment
A: probability

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** PageRank
**** Basic PageRank formula
Back: \[PR(u) = \sum_{v \in B_u} {PR(v) \over L_v}\] where \(B_u\) = pages linking to u; \(L_v\) = outgoing links from v

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** PageRank
**** How to compute PageRank
Back: initialise all pages to equal values (1/N); iteratively apply the formula until values converge

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** The Surprise-Me Button
**** Rank sink
Back: a page with no outbound links; accumulates PageRank without distributing it

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** The Surprise-Me Button
**** Why the surprise-me button
Back: without it, the surfer gets stuck on rank sinks forever

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** The Surprise-Me Button
**** Rank sink treatment
Back: rank sinks are assumed to link to all other pages in the collection

Q: * Unit 2: Search Engines
** Link Analysis and PageRank
*** The Surprise-Me Button
**** λ (lambda) — probability of clicking the surprise-me button; typically [...]
A: λ = 0.15

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** The Surprise-Me Button
**** Surprise-me algorithm
Back: -choose random r between 0 and 1
-if r < λ: jump to a random page
-if r >= λ: click a random link on the current page
-repeat

Front: * Unit 2: Search Engines
** Link Analysis and PageRank
*** The Surprise-Me Button
**** Full PageRank formula
Back: \[PR(u) = {\lambda \over N} + (1 - \lambda) \sum_{v \in B_u} {PR(v) \over L_v}\]; the λ/N term is the surprise-me button's contribution

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** What is tokenisation
Back: forming tokens (words) from a sequence of characters; terminated by spaces or punctuation

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Metadata problem
Back: web pages contain more metadata (tags, attributes, dates) than useful text; complicates tokenisation

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** What is parsing
Back: using markup tags to interpret document structure; produces a language-agnostic representation (e.g. DOM)

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Why document structure matters
Back: words in headings, bold, italic, and anchor text are more important; parser must expose this for indexing

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Two-pass tokenisation — pass 1
Back: identify and strip markup (e.g. using Beautiful Soup for HTML)

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Two-pass tokenisation — pass 2
Back: process useful words in headings, body text, and tables

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Tokenisation complication — small words
Back: 1-2 character words can be important; e.g. xp, pm, j lo

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Tokenisation complication — hyphens
Back: both hyphenated and unhyphenated forms are common; e.g. e-bay, cd-rom

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Tokenisation complication — capitalisation
Back: capitalised words can have different meanings; e.g. "Bush" vs "bush"

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Tokenisation complication — apostrophes
Back: can be part of a word, a possessive, or a mistake; e.g. o'donnell, can't, men's

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Tokenisation complication — numbers
Back: numbers including decimals can be important; e.g. Nokia 3250, QuickTime 6.5

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Tokenisation complication — periods
Back: occur in abbreviations, URLs, and sentence ends; e.g. I.B.M., Ph.D.

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Stopwords
**** What are stopwords
Back: words with little meaning in isolation ("the", "a", prepositions); discarded by text processing

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Stopwords
**** Stopword trade-off
Back: removing stopwords decreases index size and usually improves retrieval; but over-removal hurts (e.g. "to be or not to be" is all stopwords)

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Stemming
**** What is stemming (conflation)
Back: reducing different word forms to a common stem; covers inflection (swim/swam/swimming) and derivation (describe/description)

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Stemming
**** Stemming — why
Back: allows a search for "swimming" to also match "swam"

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Stemming
**** Algorithmic stemmer
Back: uses rules about word suffixes; e.g. Porter stemmer

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Stemming
**** Dictionary-based stemmer
Back: uses pre-built dictionaries of related terms; handles irregular forms (is/be/was); words don't need to look similar

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Stemming
**** Porter stemmer
Back: most popular algorithmic stemmer (1970s); multiple steps; at each step the rule for the longest applicable suffix fires

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Stemming
**** Porter step 1a — sses
Back: sses → ss (stresses → stress)

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Stemming
**** Porter step 1a — trailing s
Back: delete s if preceding part contains a vowel not immediately before the s (gaps → gap; gas → gas)

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Stemming
**** Porter step 1a — ied/ies
Back: → i if preceded by more than one letter, else ie (ties → tie; cries → cri)

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Stemming
**** Stemming impact — English
Back: small improvement (~5–10% for small collections)

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Stemming
**** Stemming impact — Arabic
Back: 50%+ improvement; highly inflected languages benefit far more

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Phrases
**** Why phrases matter
Back: more precise than single words and less ambiguous; e.g. "tropical fish" vs "fish"

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Phrases
**** POS tagger
Back: marks each word with its part-of-speech label based on context; trained on large manually-labelled corpora

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Phrases
**** POS tag — NN
Back: singular noun

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Phrases
**** POS tag — NNS
Back: plural noun

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Phrases
**** POS tag — VB
Back: base verb; VBD past tense; VBN past participle

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Phrases
**** POS tag — JJ
Back: adjective

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Phrases
**** POS tag — IN
Back: preposition

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Phrases
**** POS tag — CC
Back: conjunction (and, or)

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Phrases
**** POS tag — PRP
Back: pronoun (she, it)

Front: * Unit 2: Search Engines
** Parsing and Tokenisation
*** Phrases
**** POS tag — MD
Back: modal auxiliary (can, will)

Q: * Unit 2: Search Engines
** Indexing
*** Topical features estimate the degree to which a document is [...] a particular subject
A: about

Front: * Unit 2: Search Engines
** Indexing
*** Quality features
Back: estimate the importance and quality of a document; e.g. inbound link count, recency; independent of topical match

Front: * Unit 2: Search Engines
** Indexing
*** Query features
Back: derived from the query itself; e.g. "tropical" scores high for query "tropical fish"; related terms like "barbs" score small non-zero values

Front: * Unit 2: Search Engines
** Indexing
*** Ranking function
Back: \[R(D, Q) = \sum_i f_i(D) \cdot g_i(Q)\] where \(f_i(D)\) extracts a document feature and \(g_i(Q)\) extracts a query feature

Front: * Unit 2: Search Engines
** Indexing
*** Inverted Index
**** What is an inverted index
Back: the computational equivalent of a book's back-of-book index; associates words with documents (inverted from the usual document→words direction)

Front: * Unit 2: Search Engines
** Indexing
*** Inverted Index
**** Inverted list
Back: the list of postings for a given term

Front: * Unit 2: Search Engines
** Indexing
*** Inverted Index
**** Posting
Back: an entry in an inverted list; points to a document containing the term

Front: * Unit 2: Search Engines
** Indexing
*** Inverted Index
**** Why document numbers not URLs
Back: unique numbers are more compact for storage; a separate index maps numbers to URLs

Front: * Unit 2: Search Engines
** Indexing
*** Inverted Index
**** Inverted index type 1 — document numbers only
Back: simplest; stores only which documents contain each word

Front: * Unit 2: Search Engines
** Indexing
*** Inverted Index
**** Inverted index type 2 — word counts
Back: posting = (doc, count); distinguishes "about" from "mentions in passing"

Front: * Unit 2: Search Engines
** Indexing
*** Inverted Index
**** Inverted index type 3 — positions
Back: posting = (doc, word_position); enables exact phrase matching

Front: * Unit 2: Search Engines
** Indexing
*** Inverted Index
**** Inverted index type 4 — fields/extents
Back: adds (start, end) word-position ranges for titles/headings; enables title-weighted scoring

Front: * Unit 2: Search Engines
** Indexing
*** Inverted Index
**** Inverted index type 5 — storing scores
Back: posting = (doc, precomputed_score); fast retrieval but cannot change scoring later and loses phrase information

Front: * Unit 2: Search Engines
** Indexing
*** Inverted Index
**** Inverted index types — comparison
Back: Type | Posting | Enables | Trade-off
Doc numbers only | (doc) | Boolean retrieval | Smallest; no ranking
Word counts | (doc, count) | Frequency-based ranking | Slightly larger
Positions | (doc, pos) | Phrase matching | Much larger
Fields/extents | (doc, start, end) | Title-weighted scoring | Largest
Precomputed scores | (doc, score) | Fastest retrieval | No re-ranking; no phrases

Front: * Unit 2: Search Engines
** Indexing
*** Inverted File Storage
**** Inverted file
Back: all inverted lists stored in a single file; more efficient than one file per term

Front: * Unit 2: Search Engines
** Indexing
*** Inverted File Storage
**** One-file-per-term problem — waste
Back: file systems allocate minimum space (e.g. 1KB) per file even for a 30-byte list; 70,000 rare words × 1KB = 70MB for 2MB of data

Front: * Unit 2: Search Engines
** Indexing
*** Inverted File Storage
**** One-file-per-term problem — lookup
Back: large file directories have slow directory lookups

Q: * Unit 2: Search Engines
** Indexing
*** Inverted File Storage
**** Vocabulary / lexicon — lookup table from index terms to [...] in the inverted file; fits in a [...] loaded at startup
A: byte offsets; hash table

Q: * Unit 2: Search Engines
** Query Processing
*** Query processing goal — retrieve and rank all relevant pages; clever algorithms are [...] faster than naive ones
A: 10–100×

Front: * Unit 2: Search Engines
** Query Processing
*** Document-at-a-time (DAAT)
**** DAAT
Back: score each document completely before moving to the next; lists traversed in parallel by document number

Front: * Unit 2: Search Engines
** Query Processing
*** Document-at-a-time (DAAT)
**** DAAT — memory efficiency
Back: priority queue holds only top-k results; low scorers are evicted immediately

Front: * Unit 2: Search Engines
** Query Processing
*** Document-at-a-time (DAAT)
**** DAAT — skip non-matching
Back: only score documents that appear in at least one inverted list

Front: * Unit 2: Search Engines
** Query Processing
*** Document-at-a-time (DAAT)
**** DAAT — disadvantage
Back: switching between lists requires large list buffers in memory to avoid disk seeking

Front: * Unit 2: Search Engines
** Query Processing
*** Term-at-a-time (TAAT)
**** TAAT
Back: process all postings of one term in full, accumulate partial scores, then move to the next term

Front: * Unit 2: Search Engines
** Query Processing
*** Term-at-a-time (TAAT)
**** TAAT — disk advantage
Back: reads each inverted list sequentially; minimal disk seeking; no large list buffers needed

Front: * Unit 2: Search Engines
** Query Processing
*** Term-at-a-time (TAAT)
**** TAAT — memory disadvantage
Back: accumulator table must hold one entry per document seen; uses more memory than DAAT

Front: * Unit 2: Search Engines
** Query Processing
*** Term-at-a-time (TAAT)
**** DAAT vs TAAT — comparison
Back: Feature | DAAT | TAAT
Processing order | Document by document | Term by term
Memory | Low (priority queue, top-k only) | High (accumulator per doc seen)
Disk I/O | Higher (interleaved reads) | Lower (sequential list reads)
Best for | Memory-constrained systems | Disk-bound systems

Q: * Unit 2: Search Engines
** Query Processing
*** Conjunctive Processing and Skipping
**** Conjunctive processing — every result must contain [...] query terms; default for most web search engines
A: all

Front: * Unit 2: Search Engines
** Query Processing
*** Conjunctive Processing and Skipping
**** Conjunctive algorithm
Back: -maintain one pointer per sorted inverted list
-if da = dg: both terms present; add to results; advance both pointers
-if da < dg: advance da
-if dg < da: advance dg

Front: * Unit 2: Search Engines
** Query Processing
*** Conjunctive Processing and Skipping
**** Conjunctive — naive cost
Back: over 99% of work is wasted; e.g. "galago AND animal" scans ~300M animal postings to find ~1M matches

Front: * Unit 2: Search Engines
** Query Processing
*** Conjunctive Processing and Skipping
**** List skipping
Back: when da < dg, skip forward k documents in the longer list until sa >= dg; then search linearly within that interval

Q: * Unit 2: Search Engines
** Query Processing
*** Conjunctive Processing and Skipping
**** Skip pointers — added every c bytes; skipping reads [...] bytes instead of n (e.g. c=100, k=4 → 2.5% of data)
A: kn/c

Front: * Unit 2: Search Engines
** Query Processing
*** Conjunctive Processing and Skipping
**** Optimal skip interval — trade-off
Back: larger c shrinks pointer overhead (kn/c) but grows linear search cost (pc/2); optimum depends on p (target postings count) and is found empirically

Front: * Unit 3: Linked Data
** The Traditional Web's Limitation
*** Traditional web structure
Back: a collection of linked documents with little imposed structure; data in many formats (HTML, XML, PDF, CSV, Excel, plain text)

Front: * Unit 3: Linked Data
** The Traditional Web's Limitation
*** Traditional web — limitation
Back: formatted for human consumption; automated processes cannot easily access, search, or understand semantic relations in the data

Q: * Unit 3: Linked Data
** Linked Data
*** Linked Data — techniques for publishing and connecting [...] on the Web following W3C standards
A: structured data

Front: * Unit 3: Linked Data
** Linked Data
*** What is the Semantic Web
Back: a web of structured data entities connected by semantic relations; enables automated agents to query on behalf of users

Front: * Unit 3: Linked Data
** Linked Data
*** DBpedia
Back: extracts structured data from Wikipedia infoboxes and publishes as RDF; 228M+ entities; open and freely explorable

Front: * Unit 3: Linked Data
** Linked Data
*** LOD cloud
Back: Linked Open Data cloud; community collection of freely available linked datasets; started 2007; 1594+ datasets

Q: * Unit 3: Linked Data
** Linked Data
*** LOD cloud — governments (UK, US) contribute [...] of the data
A: 40%+

Q: * Unit 3: Linked Data
** Linked Data
*** LOD cloud — life sciences domain contributes [...] of links between datasets
A: 50%+

Q: * Unit 3: Linked Data
** Linked Data
*** LOD cloud — original data owners publish [...]; third parties publish [...]
A: one third; two thirds

Q: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF — W3C data model for Linked Data; all descriptions are statements called [...]
A: triples

Q: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF triple has three parts: [...] (entity), [...] (attribute), [...] (value); also called EAV
A: subject; predicate; object

Q: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF subject — the entity being described; must be a [...]
A: resolvable URI

Q: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF predicate — attribute of the subject; must have a [...]; abbreviated prefix:suffix (e.g. `dc:creator`)
A: resolvable URI

Q: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF object — value of the attribute; can be a [...] or a [...]
A: resolvable URI; literal

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** Why predicates use URIs
Back: resolving the URI reveals the predicate's definition; predicates may themselves be defined as RDF statements

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** dc prefix
Back: Dublin Core; `dc:creator` = shorthand for `http://purl.org/dc/elements/1.1/creator`; purl provides persistent URLs

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Prefixes
**** foaf prefix
Back: Friend of a Friend; describes people and their relationships

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Prefixes
**** vcard prefix
Back: Virtual Contact File; describes contact information

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Prefixes
**** rdfs prefix
Back: RDF Schema; vocabulary for defining classes and properties

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Prefixes
**** rdf prefix
Back: core RDF vocabulary

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** Blank Nodes
**** Blank node
Back: an anonymous RDF resource with no URI; used when minting a URI would be unnecessary

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** Blank Nodes
**** Skolemization
Back: automatically assigning URIs to blank nodes (done by some RDF databases)

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** Blank Nodes
**** Blank node advice
Back: avoid using blank nodes whenever possible

Q: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** Classes in RDF
**** RDF classes — groups of resources; membership declared with `[...]`; classes are of type `rdfs:Class`
A: rdf:type

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** Classes in RDF
**** rdfs:subClassOf
Back: states that one class is a subclass of another

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Vocabularies
**** What is an RDF vocabulary
Back: a set of term definitions for making relationships between data elements; distributed over the Web

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Vocabularies
**** FOAF
Back: Friend of a Friend — people and relationships

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Vocabularies
**** vCard vocabulary
Back: contact information

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Vocabularies
**** Dublin Core (dc)
Back: 15 elements for describing resources

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Vocabularies
**** SKOS
Back: Simple Knowledge Organization System

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Vocabularies
**** VoID
Back: Vocabulary of Interlinked Datasets

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Vocabularies
**** LOV project
Back: Linked Open Vocabularies; collected 800+ RDF vocabularies

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Vocabularies
**** Making vocabulary reuse tractable — rule 1
Back: reuse existing vocabularies whenever possible

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Vocabularies
**** Making vocabulary reuse tractable — rule 2
Back: ensure vocabulary URIs themselves follow Linked Data principles

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** URI Minting
**** URI minting rule 1
Back: use a DNS domain you control

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** URI Minting
**** URI minting rule 2
Back: use natural, human-readable keys (not `/984d6a`; use `/baked_goods/bread/rye-12`)

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** URI Minting
**** URI minting rule 3
Back: make URIs neutral to implementation details (not `/index.aspx`)

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** URI Minting
**** Fragment identifiers (#)
Back: the part of a URL after `#`; NOT sent to web servers by browsers; should be avoided in Linked Data URIs

Q: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Serialisation Formats
**** RDF is a [...], not a serialisation format; multiple formats can express the same RDF
A: data model

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Serialisation Formats
**** Turtle
Back: simple, human-readable RDF format; most common for examples

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Serialisation Formats
**** RDF/XML
Back: the original RDF format

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Serialisation Formats
**** RDFa
Back: RDF embedded in HTML attributes

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** RDF Serialisation Formats
**** JSON-LD
Back: RDF format targeting web developers

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** Turtle Syntax
**** Turtle — prefix declaration
Back: `@prefix foaf: .`

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** Turtle Syntax
**** Turtle — statement
Back: `subject predicate object .` (ends with a period)

Q: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** Turtle Syntax
**** In Turtle, a comma separates [...] for the same predicate
A: multiple objects

Q: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** Turtle Syntax
**** In Turtle, a semicolon separates [...] for the same subject
A: multiple predicates

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** Turtle Syntax
**** Turtle — `a`
Back: shorthand for `rdf:type`; e.g. `:Einstein a foaf:Person .`

Front: * Unit 3: Linked Data
** RDF — Resource Description Framework
*** Turtle Syntax
**** Turtle — example

Front: * Unit 3: Linked Data
** SPARQL
*** What is SPARQL
Back: SPARQL Protocol and RDF Query Language (recursive acronym); the query language for structured data on the Web; like SQL for RDF

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL — multiple sources
Back: can query multiple data sources at once; treats them as a single virtual RDF graph

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL vs SQL
Back: SELECT variables in SPARQL are not column names; they determine which WHERE-clause pattern variables are returned

Front: * Unit 3: Linked Data
** SPARQL
*** Key Linked Data assumption
Back: two entities sharing the same URI refer to the same thing; enables combining data from multiple sources

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Syntax
**** Basic SELECT structure
Back: `SELECT ?vars WHERE { triple patterns }`

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Syntax
**** SPARQL variable
Back: prefixed with `?`; e.g. `?entity`, `?name`, `?birth`

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Syntax
**** Triple pattern
Back: subject predicate object; any element can be a `?variable`; matched against the RDF graph

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Syntax
**** `a` in SPARQL
Back: shorthand for `rdf:type`

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Syntax
**** `[]` in SPARQL
Back: blank node; matches any subject

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Syntax
**** FROM clause
Back: query an RDF file on the web; `FROM `; multiple FROM clauses combine graphs

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Syntax
**** SPARQL endpoint
Back: web-accessible SPARQL query service; convention: at `http://{hostname}/sparql`

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Syntax
**** FILTER
Back: restricts results; `FILTER (?birth < "1900-01-01"^^xsd:date)`

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Syntax
**** ORDER BY
Back: sorts results; `ORDER BY ?name`

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Examples
**** Find entities with a foaf:name
Back: SELECT ?entity WHERE {
  ?entity foaf:name ?name
}

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Examples
**** Find subjects related to Leeds (entity)
Back: prefix dbr: <http://dbpedia.org/resource/>
SELECT ?sub WHERE { ?sub ?pred dbr:Leeds. }

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Examples
**** Find subjects related to "Leeds" (string)
Back: SELECT ?sub WHERE { ?sub ?pred "Leeds"@en. }

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Examples
**** Find names of people born in Leeds
Back: prefix dbr: <http://dbpedia.org/resource/>
SELECT ?name ?entity WHERE {
  ?entity foaf:name ?name.
  ?entity dbo:birthPlace dbr:Leeds.
}

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Examples
**** Find people born in Leeds before 1900
Back: prefix dbr: <http://dbpedia.org/resource/>
SELECT ?entity ?name ?birth WHERE {
  ?entity foaf:name ?name.
  ?entity dbo:birthPlace dbr:Leeds.
  ?entity dbo:birthDate ?birth .
  FILTER (?birth < "1900-01-01"^^xsd:date).
}

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Examples
**** People born in Berlin ordered by name
Back: SELECT ?person ?name WHERE {
  ?person dbo:birthPlace dbr:Berlin .
  ?person foaf:name ?name .
} ORDER BY ?name

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Examples
**** Capital of France
Back: SELECT ?city { dbr:France dbo:capital ?city. }

Front: * Unit 3: Linked Data
** SPARQL
*** SPARQL Examples
**** Capitals of all countries
Back: SELECT ?country ?city { ?country dbo:capital ?city. }

Front: * Cross-Cutting Themes
** REST vs the Web
Back: browsing a website IS representational state transfer; REST is a formalisation of how the WWW already works

Front: * Cross-Cutting Themes
** What makes REST hard
Back: statelessness and HATEOAS are the constraints most commonly violated in practice

Front: * Cross-Cutting Themes
** Traditional web vs Linked Data
Back: traditional web = unstructured, for humans, indexed by keyword search; Linked Data = structured RDF triples, queryable by SPARQL with semantic understanding