Structuring unstructured data, with significant granularity.
Python
0
1 commits
updated Aug 4, 2024
NuNER is one of the main data-mining components of the Panopticon platform. It contributes to finding a resolution to the problem of having to parse very large volumes of unstructured data as a continuous streaming requirement.
PLEASE NOTE: While this platform will at times heavily rely on A.I. the concept behind the implementation of tools is always the human factor, and not "A.I. for A.I.'s sake." The aim is to enhance human-driven research and investigation. At no point should anything within these repositories suggest "automation" even when wording like that may be used to accurately describe the workings of the technology. In the very best-case scenarios, we hope these tools to help prioritise and focus your work when dealing with non-human level amounts of data that have become a common aspect of the job these days, and potentially reduce the onboarding barrier of domains of interest, for which a lack of domain-knowledge may exist.
NuNER is a critical component of the Panopticon platform, designed to extract, analyze, and enrich information from unstructured text data. It's an integral part of one of the most advanced free and unrestricted investigation toolkits available today.
Panopticon is one of the most advanced investigation toolkits available to anyone who has a need for it. It is designed by drawing on the expertise of real-world practitioners, modeling their methods and approach, while identifying areas that could be optimized or otherwise enhanced by current technological advancements.
This platform empowers investigators with cutting-edge tools and techniques, streamlining complex research processes and uncovering insights that might otherwise remain hidden.
Original Concept: Initially conceived to use the NuNER model from the GLiNER family for high-performance, cost-effective named entity recognition and relationship extraction.
Current Implementation: Utilizes OpenAI's GPT models as an interim solution, ensuring continued functionality and integration within the Panopticon platform, keeping development across the board unblocked.
Future Direction: Aims to transition back to the NuNER model, combining the accuracy of large language models with the efficiency of specialized NER systems, while incorporating advanced AI-driven research and enrichment capabilities.
NuNER is tightly integrated with other components of the Panopticon platform:
NuNER is evolving to include sophisticated AI agents capable of autonomous research and data enrichment:
Gap Analysis: AI agents continuously analyze the knowledge graph to identify information gaps or inconsistencies.
Autonomous Web Research: Based on identified gaps, agents formulate queries and conduct web searches to find relevant information.
Data Extraction and Verification: New information is extracted from found sources, cross-referenced with existing data, and verified for accuracy.
Graph Integration: Verified data is seamlessly integrated into the knowledge graph, enriching existing entities and relationships.
Feedback Loop: The system learns from successful enrichments to improve future research efficiency and accuracy.
This autonomous enrichment process ensures that the knowledge graph remains up-to-date and comprehensive, providing researchers with the most current and relevant information possible.
The entire platform has an infrastructure designed to be mostly hands-off to leave more time to focus on development, or usage.
The only thing you have to configure are any API tokens of external services, and optionally you can switch to one or more alternative graph database backends.
Place the docker-compose.yml at the top-level directory that also houses the other services of the platform.
docker compose up
[Usage instructions]
[Configuration instructions]
[Visualization instructions]
Contributions to NuNER are welcome! Please refer to the main Panopticon repository for contribution guidelines.
This project is licensed under the Unlicense. This is free and unencumbered software released into the public domain. For more information, please refer to the LICENSE file in the repository or visit the Unlicense website.
For more information on the Panopticon platform and its components, including the Foucault browser extension, please visit the main Panopticon Repository.
Structuring unstructured data, with significant granularity.
Python
0
1 commits
updated Aug 4, 2024
NuNER is one of the main data-mining components of the Panopticon platform. It contributes to finding a resolution to the problem of having to parse very large volumes of unstructured data as a continuous streaming requirement.
PLEASE NOTE: While this platform will at times heavily rely on A.I. the concept behind the implementation of tools is always the human factor, and not "A.I. for A.I.'s sake." The aim is to enhance human-driven research and investigation. At no point should anything within these repositories suggest "automation" even when wording like that may be used to accurately describe the workings of the technology. In the very best-case scenarios, we hope these tools to help prioritise and focus your work when dealing with non-human level amounts of data that have become a common aspect of the job these days, and potentially reduce the onboarding barrier of domains of interest, for which a lack of domain-knowledge may exist.
NuNER is a critical component of the Panopticon platform, designed to extract, analyze, and enrich information from unstructured text data. It's an integral part of one of the most advanced free and unrestricted investigation toolkits available today.
Panopticon is one of the most advanced investigation toolkits available to anyone who has a need for it. It is designed by drawing on the expertise of real-world practitioners, modeling their methods and approach, while identifying areas that could be optimized or otherwise enhanced by current technological advancements.
This platform empowers investigators with cutting-edge tools and techniques, streamlining complex research processes and uncovering insights that might otherwise remain hidden.
Original Concept: Initially conceived to use the NuNER model from the GLiNER family for high-performance, cost-effective named entity recognition and relationship extraction.
Current Implementation: Utilizes OpenAI's GPT models as an interim solution, ensuring continued functionality and integration within the Panopticon platform, keeping development across the board unblocked.
Future Direction: Aims to transition back to the NuNER model, combining the accuracy of large language models with the efficiency of specialized NER systems, while incorporating advanced AI-driven research and enrichment capabilities.
NuNER is tightly integrated with other components of the Panopticon platform:
NuNER is evolving to include sophisticated AI agents capable of autonomous research and data enrichment:
Gap Analysis: AI agents continuously analyze the knowledge graph to identify information gaps or inconsistencies.
Autonomous Web Research: Based on identified gaps, agents formulate queries and conduct web searches to find relevant information.
Data Extraction and Verification: New information is extracted from found sources, cross-referenced with existing data, and verified for accuracy.
Graph Integration: Verified data is seamlessly integrated into the knowledge graph, enriching existing entities and relationships.
Feedback Loop: The system learns from successful enrichments to improve future research efficiency and accuracy.
This autonomous enrichment process ensures that the knowledge graph remains up-to-date and comprehensive, providing researchers with the most current and relevant information possible.
The entire platform has an infrastructure designed to be mostly hands-off to leave more time to focus on development, or usage.
The only thing you have to configure are any API tokens of external services, and optionally you can switch to one or more alternative graph database backends.
Place the docker-compose.yml at the top-level directory that also houses the other services of the platform.
docker compose up
[Usage instructions]
[Configuration instructions]
[Visualization instructions]
Contributions to NuNER are welcome! Please refer to the main Panopticon repository for contribution guidelines.
This project is licensed under the Unlicense. This is free and unencumbered software released into the public domain. For more information, please refer to the LICENSE file in the repository or visit the Unlicense website.
For more information on the Panopticon platform and its components, including the Foucault browser extension, please visit the main Panopticon Repository.