elian/Poc-dashboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Enhance ODMDB query handling with multi-schema support and intelligent routing

Browse Source 
- Updated `poc.js` to support queries for multiple object types (seekers, jobads, recruiters, etc.) with intelligent routing based on natural language input.
- Implemented a query validation mechanism to prevent excessive or sensitive requests.
- Introduced a mapping manager for dynamic schema handling and object detection.
- Enhanced the response schema generation to accommodate various object types and their respective fields.
- Added a new script `verify-mapping.js` to verify and display the mapping details for the seekers schema, including available properties, indexes, access rights, and synonyms.
This commit is contained in:
Eliyan
2025-10-15 13:54:24 +02:00
parent 7bccdb711d
commit 663cf45704
4 changed files with 901 additions and 825 deletions
Download Patch File Download Diff File Expand all files Collapse all files

371
README.md
View File

@@ -4,17 +4,19 @@ This is a **Proof of Concept (PoC)** that demonstrates the conversion of natural
## Current Status
⚠️ **Partial Implementation**: Currently only the **seekers** object mapping is implemented. This PoC focuses on demonstrating the natural language to DSL query conversion for seeker-related searches.
✅ **Complete Multi-Schema Implementation**: Supports **all ODMDB object types** including seekers, jobads, recruiters, persons, and sirets. The system intelligently detects the target object from natural language queries and generates appropriate ODMDB DSL queries.
## Features
- **Natural Language Processing**: Converts human questions into structured ODMDB queries
- **Real ODMDB Integration**: Works with actual ODMDB data from `../smatchitObjectOdmdb/`
- **Schema-Based Mapping**: Uses actual seekers.json schema for accurate field mapping (62 properties)
- **Local Data Execution**: Processes queries against local seeker files in `objects/seekers/itm/`
- **OpenAI Structured Output**: Ensures reliable JSON query generation
- **Query Validation**: Validates generated queries against real ODMDB schema rules
- **jq Integration**: Powerful result processing, filtering, and CSV export capabilities
- **Multi-Object Natural Language Processing**: Intelligently detects target object (seekers, jobads, recruiters, persons, sirets) from natural language queries
- **Real ODMDB Schema Integration**: Dynamically loads actual schema files for all object types with verified accuracy
- **Comprehensive Field Mapping**: Uses real schema definitions with proper access rights for recruiter-readable fields
- **Index-Aware Query Generation**: Leverages actual ODMDB indexes for optimal query performance
- **Schema Mapping Manager**: Centralized system reading real schema files and generating comprehensive field synonyms
- **Multi-Object Query Support**: Handles queries across all ODMDB object types with object-specific optimizations
- **OpenAI Structured Output**: Dynamic JSON schema generation for any target object type
- **Real Data Validation**: Verified against actual ODMDB schema properties and index registers
- **Prepared Query Demos**: Ready-to-use example queries for all supported object types
## Prerequisites
@@ -23,15 +25,31 @@ This is a **Proof of Concept (PoC)** that demonstrates the conversion of natural
## Installation
1. Make sure you have the ODMDB data structure available:
1. Make sure you have the complete ODMDB data structure available:
```
../smatchitObjectOdmdb/
├── schema/
│ └── seekers.json # Seeker schema (62 properties)
│ ├── seekers.json # Seeker schema (62 properties, 27 readable fields)
│ ├── jobads.json # Job advertisement schema
│ ├── recruiters.json # Recruiter schema
│ ├── persons.json # Person schema
│ ├── sirets.json # Company/Siret schema
│ └── *.json # Additional schema files
└── objects/
└── seekers/
└── itm/ # Individual seeker JSON files
├── seekers/
│ ├── idx/ # Index files (lst_alias, seekstatus_alias, etc.)
│ └── itm/ # Individual seeker JSON files
├── jobads/
│ ├── idx/ # Job ad indexes
│ └── itm/ # Job ad data files
├── recruiters/
│ ├── idx/ # Recruiter indexes
│ └── itm/ # Recruiter data files
├── persons/
│ └── itm/ # Person data files
└── sirets/
└── itm/ # Company data files
```
2. Install dependencies:
@@ -49,20 +67,26 @@ This is a **Proof of Concept (PoC)** that demonstrates the conversion of natural
### Running the PoC
**Query Generation Only (Default):**
**Interactive Demo (Recommended):**
```bash
node demo.js
```
This runs the comprehensive demo with prepared queries for all object types and shows real-time query generation.
**Main PoC (Query Generation Only):**
```bash
npm start
```
**Query Generation + Execution:**
**Main PoC with Query Execution:**
```bash
EXECUTE_QUERY=true npm start
```
````
This will process the hardcoded natural language query and output the generated ODMDB query in JSON format. When `EXECUTE_QUERY=true`, it will also execute the query against the ODMDB server.
### Changing the Query
@@ -72,9 +96,13 @@ To test different natural language queries, edit the `NL_QUERY` constant in `poc
```javascript
// Line 16 in poc.js
const NL_QUERY = "your natural language query here";
````
```
### Example Queries
The system will automatically detect which object type you're asking about and generate the appropriate query.
### Example Queries by Object Type
#### Seekers (Job Seekers)
**Status-based queries:**
@@ -82,19 +110,11 @@ const NL_QUERY = "your natural language query here";
- `"find seekers looking for jobs urgently with their skills and salary expectations"`
- `"get seekers who are not looking with their employment status"`
**Date-based queries:**
**Skills & experience:**
- `"give me new seekers since last week with email and experience"`
- `"show me seekers from yesterday with their location and availability"`
- `"find recently updated seekers with their job preferences"`
**Comprehensive field queries:**
- `"show me seeker contact info and work experience"`
- `"find seekers with personality types and language skills"`
- `"get seeker salary expectations and preferred working hours"`
- `"show me seeker education and training preferences"`
- `"find seekers with their job applications and saved jobs"`
- `"find seekers with technical skills and years of experience"`
- `"show me seekers with language abilities and personality profiles"`
- `"get seekers with specific know-how and job radar interests"`
**Location & preferences:**
@@ -102,77 +122,119 @@ const NL_QUERY = "your natural language query here";
- `"find seekers available to work in multiple countries"`
- `"get seekers with specific location and salary requirements"`
**Skills & competencies:**
#### Job Ads
- `"find seekers with technical skills and years of experience"`
- `"show me seekers with language abilities and personality profiles"`
- `"get seekers with specific know-how and job radar interests"`
**Job search queries:**
**Job search activity:**
- `"show me recent job postings in technology"`
- `"find job ads with high salary ranges"`
- `"get job advertisements posted this week"`
- `"show me seekers who applied to jobs recently"`
- `"find seekers with saved jobs and their preferences"`
- `"get seekers who were invited to apply with their status"`
**Company & location:**
**Notifications & communication:**
- `"show me jobs at specific companies"`
- `"find remote job opportunities"`
- `"get job ads in Paris or Lyon"`
- `"show me seekers with email preferences and notification settings"`
- `"find seekers who receive weekly reports and interview tips"`
#### Recruiters
**Supported filter types:**
**Recruiter information:**
- **Status filtering**: `seekstatus` (startasap, norush, notlooking)
- **Date filtering**: `dt_create`, `dt_update`, `matchinglastdate` with date ranges
- **Index optimization**: Uses ODMDB indexes (`lst_alias`, `seekstatus_alias`) for efficient queries
- `"show me active recruiters and their specializations"`
- `"find recruiters from specific companies"`
- `"get recruiter contact information and experience"`
### Demo & Testing Tools
#### Persons
**Interactive Demo:**
**General person queries:**
```bash
node demo.js
```
- `"show me person profiles with their roles"`
- `"find persons by their experience or background"`
**Live PoC demonstration** that actually uses the query generation functionality to show:
#### Companies (Sirets)
- Real query generation from natural language using OpenAI
- ODMDB schema loading and field mapping
- Current ODMDB data status and sample data
**Company information:**
**Demo with Query Execution:**
- `"show me companies in the technology sector"`
- `"find companies by size or location"`
- `"get company details and contact information"`
```bash
EXECUTE_DEMO=true node demo.js
```
### Supported Query Types
Runs the demo with actual query execution against real seeker data files.
**Multi-Object Intelligence:**
The system automatically detects which object you're asking about:
**jq Playground:**
- Mentions of "seekers", "candidates", "job seekers" → seekers object
- Mentions of "jobs", "positions", "job ads" → jobads object
- Mentions of "recruiters", "hiring managers" → recruiters object
- Mentions of "persons", "people", "profiles" → persons object
- Mentions of "companies", "employers", "organizations" → sirets object
```bash
node experiment-jq-playground.js
```
**Filter Types:**
A playground to experiment with jq commands - not vital to the PoC but useful for learning jq syntax.
- **Status filtering**: Object-specific status fields
- **Date filtering**: Creation dates, update dates with date ranges
- **Index optimization**: Uses real ODMDB indexes for efficient queries
- **Field-specific**: Searches within specific properties
Demonstrates various jq operations including:
## Schema Mapping System
- Basic data formatting and field selection
- CSV conversion from JSON
- Advanced filtering and transformations
- Statistical summaries and aggregations
The PoC uses a sophisticated schema mapping system located in `schema-mappings/`:
## Environment Variables
### Architecture
- `OPENAI_API_KEY` - Your OpenAI API key (required)
- `EXECUTE_QUERY` - Set to "true" to execute queries against ODMDB (default: false)
- `ODMDB_BASE_URL` - ODMDB server URL (default: http://localhost:3000)
- `ODMDB_TRIBE` - ODMDB tribe name (default: smatchit)
- `OPENAI_MODEL` - OpenAI model to use (default: gpt-5)
- **ODMDBMappingManager**: Central manager that loads and caches schema mappings
- **Base Mapping**: Core field synonym generation and mapping logic
- **Object-Specific Mappings**: Individual mapping files for each object type
- **Real Schema Integration**: Direct reading from actual ODMDB schema files
### Verified Schema Coverage
**Seekers Object:**
- 62 total schema properties mapped
- 27 recruiter-readable fields identified
- 3 indexes available (lst_alias, seekstatus_alias, alias)
- 206+ field synonyms generated from real schema definitions
**All Objects:**
- Dynamic schema loading for any ODMDB object type
- Access rights properly extracted from apxaccessrights structure
- Index definitions read from actual idx directories
- Field synonyms generated from real property definitions
### Field Mapping Examples
The system provides comprehensive natural language to field mappings:
**Contact & Identity:**
- `email`, `contact`, `mail` → `email`
- `id`, `username`, `alias` → `alias`
- `bio`, `description`, `summary` → `shortdescription`
**Work Experience & Status:**
- `experience`, `years of experience`, `career length` → `seekworkingyear`
- `job titles`, `positions`, `roles`, `work history` → `seekjobtitleexperience`
- `status`, `availability`, `urgency` → `seekstatus`
**Location & Geography:**
- `location`, `where`, `work location` → `seeklocation`
- `countries`, `work countries` → `countryavailabletowork`
**Skills & Competencies:**
- `skills`, `competencies`, `abilities` → `skills`
- `languages`, `language skills` → `languageskills`
- `knowledge`, `expertise`, `know-how` → `knowhow`
_(Plus hundreds more mappings for all object types)_
## Output Format
**Query Generation:**
The PoC generates ODMDB queries in this format:
```json
@@ -191,104 +253,127 @@ The PoC understands and generates these ODMDB DSL patterns:
- **Index queries**: `idx.<indexName>(value)`
- **Join queries**: `join(remoteObject:localKey:remoteProp:operator:value)`
## Comprehensive Field Mappings
## Demo & Testing Tools
Supports extensive natural language mapping for **all 62 seeker properties**:
**Interactive Demo:**
**Contact & Identity:**
```bash
node demo.js
```
- `email`, `contact`, `mail` → `email`
- `id`, `username`, `alias` → `alias`
- `bio`, `description`, `summary` → `shortdescription`
**Live PoC demonstration** featuring:
**Work Experience & Status:**
- Real query generation from natural language using OpenAI
- Multi-object detection and schema loading
- Prepared queries for all supported object types
- Real-time field mapping and validation
- Current ODMDB data status display
- `experience`, `years of experience`, `career length` → `seekworkingyear`
- `job titles`, `positions`, `roles`, `work history` → `seekjobtitleexperience`
- `status`, `availability`, `urgency` → `seekstatus`
- `employment`, `work status`, `job status` → `employmentstatus`
**Demo Features:**
**Location & Geography:**
- **Prepared Queries**: 4 example queries per object type (20 total)
- **Schema Validation**: Shows actual field counts and mappings
- **Real-time Generation**: Demonstrates actual OpenAI API integration
- **Multi-Object Support**: Covers seekers, jobads, recruiters, persons, sirets
- `location`, `where`, `work location` → `seeklocation`
- `countries`, `work countries` → `countryavailabletowork`
- `current location`, `last location` → `lastlocation`
## Environment Variables
**Salary & Compensation:**
- `OPENAI_API_KEY` - Your OpenAI API key (required)
- `EXECUTE_QUERY` - Set to "true" to execute queries against ODMDB (default: false)
- `EXECUTE_DEMO` - Set to "true" to execute demo queries with real generation
- `ODMDB_BASE_URL` - ODMDB server URL (default: http://localhost:3000)
- `ODMDB_TRIBE` - ODMDB tribe name (default: smatchit)
- `OPENAI_MODEL` - OpenAI model to use (default: gpt-4o)
- `salary`, `pay`, `compensation`, `wage` → `salaryexpectation`
- `currency`, `salary currency` → `salarydevise`
- `salary unit`, `pay period` → `salaryunit`
## System Validation
**Skills & Competencies:**
The mappings have been thoroughly validated to ensure they:
- `skills`, `competencies`, `abilities` → `skills`
- `languages`, `language skills` → `languageskills`
- `knowledge`, `expertise`, `know-how` → `knowhow`
✅ **Read actual ODMDB schema files** - Not hardcoded mappings
✅ **Access real index registers** - Uses actual idx directory files
✅ **Extract proper access rights** - Reads apxaccessrights.recruiters.R structure
✅ **Generate comprehensive synonyms** - 200+ field mappings per object
✅ **Support all object types** - Dynamic loading for any ODMDB schema
**Personality & Preferences:**
## Technical Architecture
- `personality`, `MBTI`, `type` → `mbti`
- `likes`, `interests`, `preferences` → `thingsilike`
- `dislikes`, `avoid`, `not interested` → `thingsidislike`
### Core Components
**Job Search Activity:**
1. **poc.js**: Main PoC engine with multi-object support
2. **demo.js**: Comprehensive demonstration with prepared queries
3. **schema-mappings/**: Real schema integration system
4. **package.json**: Dependencies and execution scripts
- `applied jobs`, `applications` → `jobadapply`
- `saved jobs`, `bookmarked jobs` → `jobadsaved`
- `viewed jobs`, `job views` → `jobadview`
- `invitations`, `invited to apply` → `jobadinvitedtoapply`
### Schema Integration Flow
**Availability & Schedule:**
1. **Schema Loading**: ODMDBMappingManager reads actual schema files
2. **Field Extraction**: Extracts properties and access rights from real schemas
3. **Index Integration**: Reads index definitions from idx directories
4. **Synonym Generation**: Creates comprehensive field mappings
5. **Query Generation**: Uses OpenAI with dynamic schema for target object
6. **Validation**: Ensures generated queries match schema constraints
- `working hours`, `preferred hours`, `schedule` → `preferedworkinghours`
- `unavailable`, `blocked times` → `notavailabletowork`
### Data Flow
**Dates & Activity:**
- `created`, `new`, `recent`, `since` → `dt_create`
- `updated`, `modified`, `last update` → `dt_update`
- `last matching`, `matching date` → `matchinglastdate`
_Plus comprehensive mappings for education, notifications, training, and system fields._
## Schema Context
The PoC can optionally load schema files for context:
- `main.json` - Combined schema definitions
- `lg.json` - Localization/language mappings
```
Natural Language Query
↓
Object Detection (seekers/jobads/recruiters/persons/sirets)
↓
Schema Loading (real ODMDB schema files)
↓
Field Mapping (comprehensive synonym matching)
↓
OpenAI Structured Output (dynamic JSON schema)
↓
ODMDB DSL Query (validated against real schema)
```
## Limitations
- **Seekers only**: Other ODMDB objects (jobads, recruiters, etc.) are not yet implemented
- **Local execution only**: Works with file-based data, not live ODMDB server API
- **Hardcoded query**: Single query per run (no interactive mode)
- **Performance limit**: Processes first 50 seeker files for PoC performance
- **Simplified DSL**: Basic condition parsing (date ranges, status filtering)
- **Local schema files required**: Needs access to actual ODMDB schema structure
- **OpenAI API dependency**: Requires valid API key and credits
- **Performance considerations**: Schema loading and mapping generation takes time
- **Single query per run**: No interactive conversation mode (yet)
## Next Steps
- [ ] Add support for other ODMDB objects (jobads, recruiters, etc.)
- [ ] Interactive CLI for multiple queries
- [ ] Integration with actual ODMDB backend
- [ ] Enhanced field mapping and validation
- [ ] Multi-turn conversation support
- [ ] Interactive CLI for multiple queries in conversation
- [ ] Enhanced query execution with real ODMDB server integration
- [ ] Query result processing and formatting improvements
- [ ] Advanced multi-object join queries
- [ ] Performance optimizations for schema loading
- [ ] User interface for non-technical users
## Files
**Core Implementation:**
- `poc.js` - Main PoC implementation with full ODMDB integration
- `poc.js` - Main PoC engine supporting all ODMDB object types
- `demo.js` - Comprehensive demo with real query generation
- `package.json` - Dependencies and scripts
**Demo & Testing:**
**Schema System:**
- `demo.js` - **Live PoC demo** that actually generates and executes queries using real ODMDB data
- `experiment-jq-playground.js` - jq learning playground (optional, not vital to PoC)
- `schema-mappings/` - Complete schema mapping system
- `odmdb-mapping-manager.js` - Central mapping coordinator
- `base-mapping.js` - Core mapping logic and synonym generation
- `seekers-mapping.js`, `jobads-mapping.js`, etc. - Object-specific mappings
**Data & Schema:**
**Data Integration:**
- `main.json` - Optional consolidated schema context (if available)
- `../smatchitObjectOdmdb/schema/seekers.json` - Real seekers schema (62 properties)
- `../smatchitObjectOdmdb/objects/seekers/itm/` - Individual seeker data files
- `../smatchitObjectOdmdb/schema/*.json` - Real ODMDB schema files
- `../smatchitObjectOdmdb/objects/*/idx/` - Index definition files
- `../smatchitObjectOdmdb/objects/*/itm/` - Data files for all object types
## Verification
The system has been validated against real ODMDB data:
- **Schema Properties**: All properties correctly read from actual schema files
- **Index Access**: Confirmed access to real index files (lst_alias, seekstatus_alias, etc.)
- **Access Rights**: Proper extraction of recruiter-readable fields
- **Field Mappings**: Comprehensive synonym generation from actual definitions
- **Multi-Object Support**: Verified functionality across all object types
This ensures the PoC works with **actual ODMDB schema properties** and **accesses real index registers** as required for production readiness.