One Tool, Many Industries Text Mining with Oracle Omar Alonso Chuck

One Tool, Many Industries Text Mining with Oracle Omar Alonso Chuck Adams Oracle Corp. Text Mining Summit, Boston, 2005

Agenda Introduction Text mining Define problems Present solutions A look at Oracle’s technology stack Oracle’s roadmap A case study Conclusions

Data mining and Text mining DM TM OLAP BK OLTP Keyword search Structured Data Unstructured Data Classification Clustering Ontologies NLP Inexact match

An analogy RFID and robot vision – Put tags on everything instead having the robot do the vision Similar approach for text mining – – – Language is very social, not technical Instead, start with a unified storage model Then do mining

What about text mining? Text mining is one of many features in text technology Real future of text technology is business intelligence (BI) What is BI? – Ability to make better decisions What are the obstacles today? – – Structured data is well understood Unstructured data is different

Text and XML Traditional Content Mgmt Plain Old File System File System on Steroids (WinFS) Records Mgmt, ECM XML Content Mgmt. Dynamic Doc Generation Increased exploitation of structure

First problem: access No uniform access over all sources Each source has separate storage and algebra Examples – – – – Email Databases Applications Web

Second problem: management Management of unstructured of data very poor compared with structure data Cleaning Noise is larger than in structure data Security Multilingual

Third problem – user needs Perception with current search engines Large data - 80/20 rule Doesn't provide uniform information Two users type same query and get the same results – Cricket the game or cricket the bug?

Foundations XML as the common model XML allows: – – – Manipulation data with standards Mining becomes more data mining RDF emerging as a complementary model The more structure you can explore the better you can do mining Integration use cases

Foundations - II Unstructured data is too AI Too easy to get fooled by the complexity Hybrid solution Domain knowledge – – – You know your domain You own the content You can do better

Remember?

Personalization problem Lack of personalization You own the content, you own the user Two users type the same query: “financials” – – Sales rep looks for customers and other deals Tech guy looks for bugs, architecture, etc. LDAP shows who they are Combination with query logs shows patterns in the same peer group Recommendation systems

Better Answers: Beyond Keywords Noise theory – As you cast your nets ever wider, you catch disproportionately more junk Must develop new models of Quality in the face of comprehensiveness – – Combine Link-Analysis with Context-sensitive relevance Personalization Must summarize information – Theme Maps, Gists Show patterns in information vs. many pages of hit-lists – Tree Maps, Stretch Viewer Ability to post-process and refine search hit lists – – Dynamic categories for navigation Reorder by date Progressive query relaxation – Nearest inexact match

Technology Stack Better Answers Relevance Toward BI Keyword Ranking Multi-Criterion Support Link Analysis Progressive Relaxation Query Log Analysis Classification Metadata Extraction Visualization Intelligent Match Personalization Duplicate Elimination Direct Answers

Oracle’s position Text mining is one of many tools for information retrieval and discovery in many assets Text mining is best used in the context of other techniques – – – Personalization Search query logs Visualization Product: one integrated platform

Oracle platform Integrated platform vs. niche technology Full-text searching Google, FAST XML Tamino Classification Autonomy Clustering Vivisimo Visualization Inxight Application search SAP/TREX One platform, low cost, low complexity Several products, different APIs, performance, maintenance cost, etc.

Oracle platform “If I can see further than anyone else, it is only because I am standing on the shoulders of giants” – Isaac Newton Oracle provides you all the functionality – Plus you get backup, recovery, scalability, and other benefits You build the mining application

Case study Federal customer High Performance Text Information Mining and Entity Extraction

Business Need Enterprise Search Capability Information Fusion Profiles and alerting Security – user need to know Entity identification and extraction High Performance ingestion, search, and indexing Scalability

Challenges Search quality Performance Scalability Document formats Integration Operations and maintenance

Solutions Architecture Oracle 10g Integrated Framework 10g release 2 – – Oracle Real Application Clusters Oracle Text Full text and rule based indexing Extensible thesauri Document classification Document filters – – – Oracle Partitioning Oracle Virtual Private database Oracle Advanced Security

Technical Architecture ADS LDAP Integrated for Client and Server Authentication Federated Data Access J2EE Services for mission system drill ADS OID Application Server EDL Portal User Existing Mission System EDL Portal User Application Server Process Isolated RAC DB Nodes. 1 tuned for User query and the other for data synchronization Existing Mission System Load Balancer Oracle 10g RAC EDL Portal User Interconnect Oracle 10g RAC Application Server EDL Portal User CIA PKI Authentication from ADSN clients Enterprise Meta Data Layer Scalar, Domain, and B*Tree Indices Key meta data consolidated and indexed for enterprise data layer access. Network Based Integration Hub and EDL Synchronization Services Existing Mission System

Scalable load and indexing Java Load T h re a d Java Load T h re a d D a ta C o lle c tio n P re p ro c e s s & F ilte r in g S ta n d a rd iz e d Xm l DTD U TF8 Text E x tra c te d fro m C o lle c tio n Java Load T h re a d R a w P a y lo a d Java Load D is tr ib u tio n P ro c e s s P a y lo a d In d e x Java Load T h re a d Java Load T h re a d Java Load T h re a d Java Load T h re a d Java Load T h re a d O r a c le 9 i & 9i Text S c a la r In d e x e s X M L In d e x e s

Real world results Single search for user Profiles and alerts Couple second query response 80,000,000 documents indexed 1.2 TB raw text and growing 700 Gig index size Incremental index 1-2 Gig / day

Next Steps Entity Extraction and Relationship Awareness Language specific dictionary Language specific dictionary Language specific dictionary Language specific dictionary Extracted Entities Entity identification and extraction engine Oracle 10g Text Index structure XML Interface Relationship detection engine

Oracle database 10g release 2 Enterprise Search Capability Information Fusion Profiles and alerting Security – user need to know Entity identification and extraction High Performance ingestion, search, and indexing Scalability

Conclusions Text mining is one of many features needed for BI on unstructured data – Not a silver bullet in itself Must exploit other approaches – metadata (XML, RDF), personalization, classification, entity extraction, full-text search, – Hybrid solution Focus on an integrated platform that gives you all the functionality Drive the platform for your information need