Autocrisp: AI Agents that Automate CRISP-DM

EDC support

Modern data science often follows a standard methodology called CRISP-DM (Cross Industry Standard Process for Data Mining), which outlines six essential steps: understanding the business problem, understanding the data, preparing the data, modeling, evaluating results, and deploying solutions. While powerful, this process is often time-consuming and requires advanced coding skills, posing barriers for many researchers and students. To address this, EDC started an initiative in collaboration with the Applied Data Science & AI program at Hogeschool Rotterdam where one of their students acted as lead AI engineer. This resulted in a tool called “Autocrisp”, a system that intelligently automates this entire pipeline. Built and tested in the EDC Data Sandbox, Autocrisp turns the traditional data mining process into a natural language-based experience, redefining who can participate in advanced data work.

Autocrisp uses a team of AI agents powered by Large Language Models (LLMs), designed to understand user questions and carry out complex tasks. The system is coordinated through N8N, which enables automated workflows to guide users through each phase. In the business understanding stage, users can perform meta-research, scan the web for relevant studies, and discover datasets, all with natural language prompts. As they move to data understanding and preparation, tools like Streamlit and Autogen (AG2) allow users to explore, clean, and process data. During the modeling stage, machine learning models can be built with no coding experience required. Based on the natural language prompts, the results can be evaluated. All actions are logged and reviewed in a human-in-the-loop process, where code is checked before being executed. Support systems like Supabase handle secure data storage and user authentication behind the scenes.