MarkLogic Query and Search - Revision history

Bowersmt: Fixed category

2014-11-20T04:38:17Z

Fixed category

← Older revision		Revision as of 04:38, 20 November 2014
Line 155:		Line 155:
	* [http://developer.marklogic.com/blog/grokking-the-cts-api "Grokking the cts API"] is a great overview of MarkLogic's search and query capabilities.		* [http://developer.marklogic.com/blog/grokking-the-cts-api "Grokking the cts API"] is a great overview of MarkLogic's search and query capabilities.

	~~<nowiki>~~[[Category: MarkLogic]]~~</nowiki>~~		[[Category: MarkLogic]]

Bowersmt at 04:37, 20 November 2014

2014-11-20T04:37:15Z

Show changes

Bowersmt: Updated cts:not-query

2014-11-19T07:01:10Z

Updated cts:not-query

← Older revision		Revision as of 07:01, 19 November 2014
Line 154:		Line 154:

	*'''<code>[http://docs.marklogic.com/cts:not-query cts:not-query]</code>''' returns a query that subtracts one or more sub-queries.		*'''<code>[http://docs.marklogic.com/cts:not-query cts:not-query]</code>''' returns a query that subtracts one or more sub-queries.
	**It works by taking the list of documents in the database and subtracting documents returned from its sub-queries. It returns all remaining documents. It does not filter documents returned from its sub-queries because filtering happens after all index resolution has completed, and <code>cts:not-query</code) executes during index resolution.		**It works by taking the list of all documents in the database and subtracting the documents returned from its sub-queries. It returns all remaining documents. It does not filter documents returned from its sub-queries because filtering happens after all index resolution has completed, and <code>cts:not-query</code> executes during index resolution. Thus, only if its sub-queries return accurate results (with no false-positives), then the results of <code>cts:not-query</code> are accurate. Thus, it only produces accurate results when all its sub-queries are resolvable completely from indexes with no filtering required.
	**'''Warning:''' cts:not-query can be inaccurate. It can be missing documents when ~~any~~ of its sub-queries ~~requires~~ filtering to be ~~accurate. It~~ only ~~produces accurate results when all its sub-queries are resolvable completely from~~ indexes ~~with no filtering required~~.		**'''Warning:''' cts:not-query can be inaccurate. It can be missing some documents. This happens when one or more of its sub-queries require filtering (i.e. can't be resolved accurately using only indexes). When sub-queries require filtering, they may return false-positive matches. A false-positive match occurs when the indexes cannot determine for sure if a document matches; so MarkLogic includes the document just in case. As the last step in a query, MarkLogic opens all potentially matching documents and filters out false-positive matches. Thus, false-positives are not a match, they are a failed potential match. False positives can be prevented by writing sub-queries that are resolved completely by the indexes so that there is no need for filtering.
	~~***~~When sub-queries require filtering, they may return false-positive matches. A false-positive match occurs when the indexes cannot determine for sure if a document matches so MarkLogic includes it just in case. As the last step in a query, MarkLogic opens all ~~the~~ potentially matching documents ~~returned by the indexes~~ and filters out ~~any~~ false-positive matches.		***Because <code>cts:not-query</code> runs during index resolution (before filtering occurs), it subtracts out any false-positive matches returned by its sub-queries. False-positives should not be subtracted out. But MarkLogic doesn't know which documents are false-positives until filtering, and it can't do filtering during <code>cts:not-query</code> because it is in the middle of doing index resolution. Thus, MarkLogic subtracts out false-positives and this causes <code>cts:not-query</code> to return too few documents. If we prevent sub-queries from returning false-positives, <code>cts:not-query</code> returns accurate results.
	***~~When~~ <code>cts:not-query</code> subtracts its sub-queries ~~from the~~ documents ~~in the database~~, it ~~also subtracts false~~-~~positive matches~~ because it ~~runs during~~ index resolution ~~and before filtering occurs~~. ~~If there are~~ false-~~positive matches, then~~ <code>cts:not-query</code> ~~won't include them~~. If ~~the~~ sub-queries ~~return inaccurate false-positive matches, then they get subtracted out of the results, which creates inaccurate results. When the sub-queries return accurate results (with no~~ false-positives), ~~then the results of~~ <code>cts:not-query</code> ~~are~~ accurate.

	*'''<code>[http://docs.marklogic.com/cts:and-not-query cts:and-not-query]</code>'''		*'''<code>[http://docs.marklogic.com/cts:and-not-query cts:and-not-query]</code>'''

Bowersmt: Updated cts:not-query

2014-11-19T06:18:40Z

Updated cts:not-query

← Older revision		Revision as of 06:18, 19 November 2014
Line 138:		Line 138:
	***Searching attributes, text nodes, and childless elements is similar to searching documents and elements with children — except the cts query cannot look for nested structures.		***Searching attributes, text nodes, and childless elements is similar to searching documents and elements with children — except the cts query cannot look for nested structures.

	*'''<code>[http://docs.marklogic.com/cts:contains cts:contains]</code>''' is the second most important function because it returns true when a cts query finds at least one matching node in the nodes returned by an XPath expression (or a sequence of values). In other words, it returns true when at least one of the specified nodes '''contains''' the precise combination of words, phrases, values, structures, and/or triples in a cts query.		*'''<code>[http://docs.marklogic.com/cts:contains cts:contains]</code>''' is the second most important function because it returns true when a cts query finds at least one matching node in the nodes returned by an XPath expression (or a sequence of values). In other words, it returns true when at least one of the specified nodes '''contains''' the precise combination of words, phrases, values, structures, and/or triples in a cts query.
			**NOTE: in the second example for <code>[http://docs.marklogic.com/cts:not-query cts:not-query]</code>, it says cts:contains forces the constraint to happen in the filtering stage of the query. Is this always true for cts:contains, or is it only true in the example?

	=== Constructing a Query ===		=== Constructing a Query ===
Line 153:		Line 154:

	*'''<code>[http://docs.marklogic.com/cts:not-query cts:not-query]</code>''' returns a query that subtracts one or more sub-queries.		*'''<code>[http://docs.marklogic.com/cts:not-query cts:not-query]</code>''' returns a query that subtracts one or more sub-queries.
			**It works by taking the list of documents in the database and subtracting documents returned from its sub-queries. It returns all remaining documents. It does not filter documents returned from its sub-queries because filtering happens after all index resolution has completed, and <code>cts:not-query</code) executes during index resolution.
			**'''Warning:''' cts:not-query can be inaccurate. It can be missing documents when any of its sub-queries requires filtering to be accurate. It only produces accurate results when all its sub-queries are resolvable completely from indexes with no filtering required.
			***When sub-queries require filtering, they may return false-positive matches. A false-positive match occurs when the indexes cannot determine for sure if a document matches so MarkLogic includes it just in case. As the last step in a query, MarkLogic opens all the potentially matching documents returned by the indexes and filters out any false-positive matches.
			***When <code>cts:not-query</code> subtracts its sub-queries from the documents in the database, it also subtracts false-positive matches because it runs during index resolution and before filtering occurs. If there are false-positive matches, then <code>cts:not-query</code> won't include them. If the sub-queries return inaccurate false-positive matches, then they get subtracted out of the results, which creates inaccurate results. When the sub-queries return accurate results (with no false-positives), then the results of <code>cts:not-query</code> are accurate.

	*'''<code>[http://docs.marklogic.com/cts:and-not-query cts:and-not-query]</code>'''		*'''<code>[http://docs.marklogic.com/cts:and-not-query cts:and-not-query]</code>'''

Bowersmt: Clarified how cts:search works

2014-11-19T04:49:10Z

Clarified how cts:search works

← Older revision		Revision as of 04:49, 19 November 2014
Line 79:		Line 79:
	When documents are inserted or updated, MarkLogic indexes them in RAM (as well as appends the changes to a journal on disk). When documents in RAM start to consume too much RAM, MarkLogic writes them to disk as a "stand" in a "forest" and starts creating a new "stand" in RAM.		When documents are inserted or updated, MarkLogic indexes them in RAM (as well as appends the changes to a journal on disk). When documents in RAM start to consume too much RAM, MarkLogic writes them to disk as a "stand" in a "forest" and starts creating a new "stand" in RAM.

	When you query or search for documents, MarkLogic goes to the indexes in the stands (which are cached in RAM) and evaluates the query in parallel. Each stand index returns a list of document IDs (a list of numbers) that match the query. MarkLogic takes the document IDs returned by each stand and merges them into a final list of document IDs, ~~whcih~~ it uses to retrieve the documents from cache or disk. This works well because computers are very fast at sort merging lists of numbers.		When you query or search for documents, MarkLogic goes to the indexes in the stands (which are cached in RAM) and evaluates the query in parallel. Each stand index returns a list of document IDs (a list of numbers) that match the query. MarkLogic takes the document IDs returned by each stand and merges them into a final list of document IDs, which it uses to retrieve the documents from cache or disk. This works well because computers are very fast at sort merging lists of numbers.

	MarkLogic uses the same indexes for both queries and searches. This allows you to combine search and query expressions. The main difference between search and query is how documents are sorted: searches sort by relevance and queries sort by values. Another difference is that queries tend to use value indexes and searches tend to use word and phrase indexes.		MarkLogic uses the same indexes for both queries and searches. This allows you to combine search and query expressions. The main difference between search and query is how documents are sorted: searches sort by relevance and queries sort by values. Another difference is that queries tend to use value indexes and searches tend to use word and phrase indexes.
Line 123:		Line 123:
	=== Executing a Search ===		=== Executing a Search ===
	*'''<code>[http://docs.marklogic.com/cts:search cts:search]</code>''' is the most important function because it executes a search and returns matching nodes.		*'''<code>[http://docs.marklogic.com/cts:search cts:search]</code>''' is the most important function because it executes a search and returns matching nodes.
	**'''You must include an XPath expression''' to define which nodes to ''search'' and ''return''.		**'''You must include an XPath expression''' to define which nodes to ''search'' within and to ''return''.
	**The search returns ''nodes'', which can be documents, elements, attributes, or text. Thus, a search doesn't have to return entire documents. Depending on the scope of your XPath statement, it may return entire documents, or one matching node from each document, or many matching nodes from each document.		***The search returns ''nodes'', which can be documents, elements, attributes, or text. Thus, a search doesn't have to return entire documents. Depending on the scope of your XPath statement, it may return entire documents, or one matching node from each document, or many matching nodes from each document.
	**'''You must include a cts query expression''' to filter the contents of ''nodes'' selected by the XPath expression. This allows you to search for specific content within specific nodes and return only those nodes that contain that content. If you want to return all the nodes selected by the XPath expression, you can use the empty query expression: <code>cts:and-query(())</code>.		**'''You must include a cts query expression''' to filter the contents of ''nodes'' selected by the XPath expression. This allows you to search for specific content within specific nodes and return only those nodes that contain that content.
			***MarkLogic limits search results to words, phrases, values, and structures in the query that occur within the nodes specified by the XPath expression.
			***If you want to return entire documents and search within specific elements and attributes, then the XPath expression should specify the root element and the query should use the element and element-attribute queries to limit where the search occurs.
			***If you want to return all the nodes selected by the XPath expression, you can use the empty query expression: <code>cts:and-query(())</code>.
	**You may limit the search to specific forests. You may set options like unfiltered, faceted, unchecked, quality weight, relevance scoring method, and relevance trace.		**You may limit the search to specific forests. You may set options like unfiltered, faceted, unchecked, quality weight, relevance scoring method, and relevance trace.
	**Use Cases:		**Use Cases:
Line 143:		Line 146:
	The composite query constructors build up new queries from other queries, and queries can be nested within queries.		The composite query constructors build up new queries from other queries, and queries can be nested within queries.

	*'''<code>[http://docs.marklogic.com/cts:and-query cts:and-query]</code>''' returns a query that intersects two or more sub-queries. You may optionally specify an "ordered" option that requires document contents to match sub-queries in the order they are listed; for example if the sub-queries are "To be" and "or not to be", then the query will match documents that have the phrase "To be" ''before'' "or not to be". The default is "unordered", which allows each sub-query to match the content in any order that it occurs in the document. If you pass in the empty sequence, such as <code>cts:and-query(())</code>, then it will match every document in the database.		*'''<code>[http://docs.marklogic.com/cts:and-query cts:and-query]</code>''' returns a query that intersects two or more sub-queries.
			**You may optionally specify an "ordered" option that requires document contents to match sub-queries in the order they are listed; for example if the sub-queries are "To be" and "or not to be", then the query will match documents that have the phrase "To be" ''before'' "or not to be". The default is "unordered", which allows each sub-query to match the content in any order that it occurs in the document.
			**If you pass in the empty sequence, such as <code>cts:and-query(())</code>, then it will match every document in the database.

	*'''<code>[http://docs.marklogic.com/cts:or-query cts:or-query]</code>''' returns a query that unions two or more sub-queries.		*'''<code>[http://docs.marklogic.com/cts:or-query cts:or-query]</code>''' returns a query that unions two or more sub-queries.

	*'''<code>[http://docs.marklogic.com/cts:not-query cts:not-query]</code>'''		*'''<code>[http://docs.marklogic.com/cts:not-query cts:not-query]</code>''' returns a query that subtracts one or more sub-queries.

	*'''<code>[http://docs.marklogic.com/cts:and-not-query cts:and-not-query]</code>'''		*'''<code>[http://docs.marklogic.com/cts:and-not-query cts:and-not-query]</code>'''

Bowersmt: Minor changes

2014-11-18T15:21:28Z

Minor changes

← Older revision		Revision as of 15:21, 18 November 2014
Line 143:		Line 143:
	The composite query constructors build up new queries from other queries, and queries can be nested within queries.		The composite query constructors build up new queries from other queries, and queries can be nested within queries.

	*'''<code>[http://docs.marklogic.com/cts:and-query cts:and-query]</code>''' returns a query that intersects two or more sub-queries.		*'''<code>[http://docs.marklogic.com/cts:and-query cts:and-query]</code>''' returns a query that intersects two or more sub-queries. You may optionally specify an "ordered" option that requires document contents to match sub-queries in the order they are listed; for example if the sub-queries are "To be" and "or not to be", then the query will match documents that have the phrase "To be" ''before'' "or not to be". The default is "unordered", which allows each sub-query to match the content in any order that it occurs in the document. If you pass in the empty sequence, such as <code>cts:and-query(())</code>, then it will match every document in the database.

	*'''<code>[http://docs.marklogic.com/cts:or-query cts:or-query]</code>'''		*'''<code>[http://docs.marklogic.com/cts:or-query cts:or-query]</code>''' returns a query that unions two or more sub-queries.

	*'''<code>[http://docs.marklogic.com/cts:not-query cts:not-query]</code>'''		*'''<code>[http://docs.marklogic.com/cts:not-query cts:not-query]</code>'''

Bowersmt: Added more information about searching

2014-11-15T22:54:25Z

Added more information about searching

Bowersmt: Added more info on contains()

2014-11-15T18:02:20Z

Added more info on contains()

← Older revision		Revision as of 18:02, 15 November 2014
Line 5:		Line 5:

	=== How does MarkLogic compare to a relational database? ===		=== How does MarkLogic compare to a relational database? ===
	A document in MarkLogic is very much like a row in a relational database, and a document type ties documents together like a table ties rows together. Both use indexes to resolve queries without having to scan through rows or documents.		A document in MarkLogic is very much like a row in a relational database, and a document type ties documents together like a table ties rows together. MarkLogic collections group documents together, and this is a little bit like views in a relational database, which group together a filtered set of rows from one or more tables. A document in MarkLogic can have a flat structure like a relational table, or it can contain complex nestings of data. Each nesting of data in a document represents another table in a relational database. For example, a single document in MarkLogic may represent many relational tables and its nested structure represents their one-to-one, many-to-one, and many-to-many relationships. Both MarkLogic and relational databases use indexes to resolve queries without having to scan through rows or documents.

	Unlike relational databases, MarkLogic can't do full table scans; queries must use indexes. In contrast, relational databases often use a full table scan to retrieve every row in a table. A rule of thumb in relational databases is that it is faster to directly read in ''all'' the rows when a query accesses more than roughly 25% of the rows. This is because relational databases use variations of the B-Tree index, which requires 3-4 random IOs to find one row and 1 IO to retrieve the row: this is 4-5 total IOs per row. B-Tree indexes work best for single row lookups, but they become increasingly costly the more rows they retrieve. If an index has to lookup 25% of its rows with 4 IOs per row, the database will do the same amount of IO if it reads in all of the rows (1 IO per row). Since sequential IO on disks is faster than random IO, relational databases prefer full table scans when accessing more than about 25% of rows in a table. On analytical data warehouses, full table scans are always preferred.		Unlike relational databases, MarkLogic can't do full table scans; queries must use indexes. In contrast, relational databases often use a full table scan to retrieve every row in a table. A rule of thumb in relational databases is that it is faster to directly read in ''all'' the rows when a query accesses more than roughly 25% of the rows. This is because relational databases use variations of the B-Tree index, which requires 3-4 random IOs to find one row and 1 IO to retrieve the row: this is 4-5 total IOs per row. B-Tree indexes work best for single row lookups, but they become increasingly costly the more rows they retrieve. If an index has to lookup 25% of its rows with 4 IOs per row, the database will do the same amount of IO if it reads in all of the rows (1 IO per row). Since sequential IO on disks is faster than random IO, relational databases prefer full table scans when accessing more than about 25% of rows in a table. On analytical data warehouses, full table scans are always preferred.
Line 118:		Line 118:
	*'''<code>[http://docs.marklogic.com/cts:search cts:search]</code>''' is the most important function because it executes a search and returns matching nodes.		*'''<code>[http://docs.marklogic.com/cts:search cts:search]</code>''' is the most important function because it executes a search and returns matching nodes.
	**'''You must include an XPath expression''' to define which nodes to ''search'' and ''return''.		**'''You must include an XPath expression''' to define which nodes to ''search'' and ''return''.
	**The search returns ''nodes'', which can be documents, elements, attributes, text~~, processing instructions, comments, and namespaces~~. Thus, a search doesn't have to return entire documents. Depending on the scope of your XPath statement, it may return entire documents, or one matching node from each document, or many matching nodes from each document.		**The search returns ''nodes'', which can be documents, elements, attributes, or text. Thus, a search doesn't have to return entire documents. Depending on the scope of your XPath statement, it may return entire documents, or one matching node from each document, or many matching nodes from each document.
	**'''You must include a cts query expression''' to filter the contents of ''nodes'' selected by the XPath expression. This allows you to search for specific content within specific nodes and return only those nodes that contain that content. If you want to return all the nodes selected by the XPath expression, you can use the empty query expression: <code>cts:and-query(())</code>.		**'''You must include a cts query expression''' to filter the contents of ''nodes'' selected by the XPath expression. This allows you to search for specific content within specific nodes and return only those nodes that contain that content. If you want to return all the nodes selected by the XPath expression, you can use the empty query expression: <code>cts:and-query(())</code>.
	**You may limit the search to specific forests. You may set options like unfiltered, faceted, unchecked, quality weight, relevance scoring method, and relevance trace.		**You may limit the search to specific forests. You may set options like unfiltered, faceted, unchecked, quality weight, relevance scoring method, and relevance trace.
Line 125:		Line 125:
	***A search can extract and return titles from documents as long as the contents of the titles match the XPath and query expressions.		***A search can extract and return titles from documents as long as the contents of the titles match the XPath and query expressions.
	***A search can extract and return all the paragraphs from documents as long as the contents of the paragraphs match the XPath and query expressions.		***A search can extract and return all the paragraphs from documents as long as the contents of the paragraphs match the XPath and query expressions.
			**Notes:
			***There is no functional difference between searching entire documents and elements because an element has the same structure as document. Both may contain complex nested elements or both may simply contain a value. In both cases, MarkLogic retrieves entire documents. In the case of elements, it extracts the requested elements from each retrieved document.
			***Searching attributes, text nodes, and childless elements is similar to searching documents and elements with children — except the cts query cannot look for nested structures.

	*'''<code>[http://docs.marklogic.com/cts:contains cts:contains]</code>''' is the second most important function because it returns true when ~~any of~~ a sequence of ~~documents or~~ values contains words, phrases, values, structures or triples ~~specified by~~ a cts query.		*'''<code>[http://docs.marklogic.com/cts:contains cts:contains]</code>''' is the second most important function because it returns true when a cts query finds at least one matching node in the nodes returned by an XPath expression (or a sequence of values). In other words, it returns true when at least one of the specified nodes '''contains''' the precise combination of words, phrases, values, structures, and/or triples in a cts query.

	=== Constructing a Query ===		=== Constructing a Query ===

Bowersmt: Added some search functions

2014-11-15T07:12:09Z

Added some search functions

@@ Line 79: / Line 79: @@
 === What specific indexes does MarkLogic have? ===
 MarkLogic uses indexes to extract words, values, structures, and links out of documents. An index is an altered view of a document. It enables you to search and query for documents <br>'''as if they contained only:'''
 *'''Words:''' a flat set of words without structure, such as <br><code>["Mary", "had", "a", "little", "lamb"]</code>
 *'''Phrases:''' a flat set of phrases without structure, such as <br><code>["Mary had", "had a", "a little", "little lamb"]</code>
 *'''Elements:''' a flat set of elements without values, such as <br><code>["poem", "text", "line"]</code>
 *'''Element-values:''' a flat set of elements with string values, such as <br><code>"line": "Mary had a little lamb"</code>
 *'''Element-words:''' a flat set of elements with words, such as <br><code>"line": ["Mary", "had", "a", "little", "lamb"]</code>
 *'''Element-phrases:''' a flat set of elements with phrases, such as <br><code>"line": ["Mary had", "had a", "a little", "little lamb"]</code>
 *'''Element-value lexicon:''' a flat set of elements with typed values, such as <br><code>"lineSequence": 3</code><br> or a list, count, or co-occurrence of element values scoped to a database, set of documents, or one document.
 *'''XPath-structure-value lexicon:''' a flat set of hierarchical elements with typed values, such as <br><code>poem.text.line.lineSequence: 3</code><br> or a list, count, or co-occurrence of hierarchical element values scoped to a database, set of documents, or one document.
 *'''XPath-structure:'''hierarchical structures without values, such as <br><code>poem.text.line</code>
 *'''XPath-structure-words:''' hierarchical structures with words, such as <br><code>poem.text.line: ["Mary", "had", "a", "little", "lamb"]</code>
 *'''XPath-structure-phrases:''' hierarchical structures with phrases, such as <br><code>poem.text.line: ["Mary had", "had a", "a little", "little lamb"]</code>
 *'''XPath-structure-values:''' hierarchical structures with values, such as <br><code>poem.text.line: "Mary had a little lamb"</code>
 *'''RDF document links''' to other documents, such as <br><code>{"subject": "http://example.org/documents/thisDoc", <br>"predicate": "http://example.org/predicates/relatesSomehowTo", <br>"object": "http://example.org/documents/thatDoc"}</code>
 *'''RDF abstract links''' to other abstract concepts, such as <br><code>{"subject": "http://example.org/documents/thisDoc", <br>"predicate": "http://example.org/predicates/relatesSomehowTo", <br>"object": "http://example.org/documents/someConceptWithNoDocumentAtTheURI"}</code>
 *'''RDF data links''' to data, such as <br><code>{"subject": "http://example.org/documents/thisDoc", <br>"predicate": "http://example.org/predicates/ageInYears", <br>"object": 12}</code>
@@ Line 98: / Line 113: @@
 Search and query expressions are fully composable; i.e. they can be nested inside each other and combined using AND, OR, and NOT expressions.
 == Training Resources for MarkLogic Search ==

Bowersmt: Added Back to MarkLogic

2014-11-14T08:54:41Z

Added Back to MarkLogic

← Older revision		Revision as of 08:54, 14 November 2014
Line 1:		Line 1:
	== MarkLogic ~~Query and Search~~ ==		[[MarkLogic\|« Back to MarkLogic]]

			== Searching and Querying MarkLogic ==
	Indexes are central to everything you do in MarkLogic. They are central to how MarkLogic stores data, scales massively, and processes data quickly. They are central to how you develop because you have to use MarkLogic's indexes to query and search documents. You have to understand how MarkLogic's indexes work to be able to create fast queries. And you have to know how to model document structure to work best with indexes.		Indexes are central to everything you do in MarkLogic. They are central to how MarkLogic stores data, scales massively, and processes data quickly. They are central to how you develop because you have to use MarkLogic's indexes to query and search documents. You have to understand how MarkLogic's indexes work to be able to create fast queries. And you have to know how to model document structure to work best with indexes.

← Older revision		Revision as of 22:54, 15 November 2014
Line 5:		Line 5:

	=== How does MarkLogic compare to a relational database? ===		=== How does MarkLogic compare to a relational database? ===
	A document in MarkLogic is very much like a row in a relational database, and a document type ties documents together like a table ties rows together. MarkLogic collections group documents together~~, and this is a little bit~~ like views in a relational database, which group together a filtered set of rows from one or more tables. A document in MarkLogic can have a flat structure like a relational table, or it can contain complex nestings of data. Each nesting of data in a document represents another table in a relational database. For example, a single document in MarkLogic may represent many relational tables and ~~its~~ nested ~~structure~~ represents their one-to-one, many-to-one, and many-to-many relationships. Both MarkLogic and relational databases use indexes to resolve queries without having to scan through rows or documents.		A document in MarkLogic is very much like a row in a relational database, and a document type ties documents together like a table ties rows together. MarkLogic collections group documents together in any way you want. Collections are like views in a relational database, which group together a filtered set of rows from one or more tables, but collections do not join and merge document content and views cannot arbitrarily include any row from any table. A document in MarkLogic can have a flat structure like a relational table, or it can contain complex nestings of data. Each nesting of data in a document represents another table in a relational database. For example, a single document in MarkLogic may represent many relational tables, and how they are nested represents their one-to-one, many-to-one, and many-to-many relationships.

			Both MarkLogic and relational databases use indexes to resolve queries without having to scan through all documents or rows. MarkLogic indexes are designed to find any word, phrase, value, element, or element structure -- no matter where it occurs in document structure, no matter how many times it occurs within a document, and no matter what type of document contains it. For example, you can search for and return a <title> element and MarkLogic will return the contents of all <title> elements it finds — even if <title> elements are in different types of documents. This is similar to doing a union query across multiple tables in relational databases.

			MarkLogic indexes also let you search for words, phrases, values, elements, or element structures in very specific locations, such as in only certain types of documents or within certain document structures, or in specific collections or folders. For example, you can search for and return all <title> elements that are found in the header section of poetry documents in the Shakespeare and Tennyson collections.

			MarkLogic is not designed for joining documents like a relational database. A relational join searches for matching values across two or more tables and then merges matching rows into a single row. This can be done in MarkLogic, but you have to write advanced code to iterate through query results to match values across different document types, and merge matching documents into a single document. Also since MarkLogic documents are often nested structures, you can't simply merge them, you have to copy nested content from one document into a specific nested location in another.

	Unlike relational databases, MarkLogic can't do full table scans; queries must use indexes. In contrast, relational databases often use a full table scan to retrieve every row in a table. A rule of thumb in relational databases is that it is faster to directly read in ''all'' the rows when a query accesses more than roughly 25% of the rows. This is because relational databases use variations of the B-Tree index, which requires 3-4 random IOs to find one row and 1 IO to retrieve the row: this is 4-5 total IOs per row. B-Tree indexes work best for single row lookups, but they become increasingly costly the more rows they retrieve. If an index has to lookup 25% of its rows with 4 IOs per row, the database will do the same amount of IO if it reads in all of the rows (1 IO per row). Since sequential IO on disks is faster than random IO, relational databases prefer full table scans when accessing more than about 25% of rows in a table. On analytical data warehouses, full table scans are always preferred.		Unlike relational databases, MarkLogic can't do full table scans; queries must use indexes. In contrast, relational databases often use a full table scan to retrieve every row in a table. A rule of thumb in relational databases is that it is faster to directly read in ''all'' the rows when a query accesses more than roughly 25% of the rows. This is because relational databases use variations of the B-Tree index, which requires 3-4 random IOs to find one row and 1 IO to retrieve the row: this is 4-5 total IOs per row. B-Tree indexes work best for single row lookups, but they become increasingly costly the more rows they retrieve. If an index has to lookup 25% of its rows with 4 IOs per row, the database will do the same amount of IO if it reads in all of the rows (1 IO per row). Since sequential IO on disks is faster than random IO, relational databases prefer full table scans when accessing more than about 25% of rows in a table. On analytical data warehouses, full table scans are always preferred.
Line 18:		Line 24:

	A client contacts one of the servers in a MarkLogic cluster and initiates a query or search. Because MarkLogic communicates through REST web services, a load balancer can spread requests across the servers in a MarkLogic cluster. The initiating server does the following		A client contacts one of the servers in a MarkLogic cluster and initiates a query or search. Because MarkLogic communicates through REST web services, a load balancer can spread requests across the servers in a MarkLogic cluster. The initiating server does the following
	#it validates the query request from the client		#It validates the query request from the client
	#it parses and optimizes the query		#It parses and optimizes the query
	#it sends the query to all servers that have forests in the database		#It sends the query to all servers that have forests in the database
	##~~each~~ server sends it to its forests		##Each server sends it to its forests
	##~~each~~ forest sends it to each stand		##Each forest sends it to each stand
	##~~each~~ stand uses its indexes to find matching documents		##Each stand uses its indexes to find matching documents
	###~~all~~ stands and forests query the indexes in parallel		###All stands and forests query the indexes in parallel
	##~~forests~~ combine the matching document IDs from all their stands		##Forests combine the matching document IDs from all their stands
	##~~forests~~ sort the matching document IDs (in relevance score order or value order)		##Forests sort the matching document IDs (in relevance score order or value order)
	##~~when~~ complete, each forest returns a sorted iterator to the initiating server		##When complete, each forest returns a sorted iterator to the initiating server
	#~~when~~ each forest has returned an iterator		#When each forest has returned an iterator
	#it walks the sorted iterators from all the forests and combines the results in sorted		#It walks the sorted iterators from all the forests and combines the results in sorted
	#it retrieves matching documents from its ''Expanded Tree Cache'' or from the forests when documents are not cached		#It retrieves matching documents from its ''Expanded Tree Cache'' or from the forests when documents are not cached
	##forests retrieve documents from their ''Compressed Tree Cache'' or from disk when documents are not cached		##forests retrieve documents from their ''Compressed Tree Cache'' or from disk when documents are not cached
	#it filters returned documents to remove false positive matches		#It filters returned documents to remove false positive matches
	#it optionally transforms documents		#It optionally transforms documents
	##it creates new documents by extracting matching elements, changing structure, changing file format, etc.		##It creates new documents by extracting matching elements, changing structure, changing file format, etc.
	#it returns all matching (optionally transformed) documents to the requester		#It returns all matching (optionally transformed) documents to the requester
	#it saves resulting document IDs when paging query results, so it can return subsequent pages of matching documents in subsequent requests		#It saves resulting document IDs when paging query results, so it can return subsequent pages of matching documents in subsequent requests

	The initiating server's process is CPU intensive, serialized, and blocks while waiting on forests. To prevent this from being a bottleneck to the cluster, you can dedicate nodes in the cluster to do nothing but evaluate queries. These are called e-nodes (evaluation nodes). These nodes don't hold data. If they held data, the ''serialized'' evaluation process would compete with the ''parallel'' index matching process. Instead e-nodes are dedicated to validating, parsing, sorting, filtering, transforming, and paginating documents. They cache documents in the ''Expanded Tree Cache'' to minimize the number of documents they have to retrieve from data nodes (d-nodes). D-nodes only do parallel index processing and sorting. They cache documents in their ''Compressed Tree Cache'' to reduce disk IO.		The initiating server's process is CPU intensive, serialized, and blocks while waiting on forests. To prevent this from being a bottleneck to the cluster, you can dedicate nodes in the cluster to do nothing but evaluate queries. These are called e-nodes (evaluation nodes). These nodes don't hold data. If they held data, the ''serialized'' evaluation process would compete with the ''parallel'' index matching process. Instead e-nodes are dedicated to validating, parsing, sorting, filtering, transforming, and paginating documents. They cache documents in the ''Expanded Tree Cache'' to minimize the number of documents they have to retrieve from data nodes (d-nodes). D-nodes only do parallel index processing and sorting. They cache documents in their ''Compressed Tree Cache'' to reduce disk IO.