Questions tagged [dataimporthandler]
The DataImportHandler is a Solr contrib that provides a configuration driven way to import data into Solr, supporting both full and incremental (delta) import.
398
questions
0
votes
0answers
10 views
Solr DIH, optional value in field with DateFormatTransformer o RegexTransformer
I have a Solr 8.7 managed schema which uses a DB as a source, I've configured my db-data-config.xml with all the fields.
I have a column in db which is a date, in yyyyMMdd format (for example the 18 ...
0
votes
1answer
27 views
Solr not returning all documents after importing with the Data Handler
I have a Solr 8.7.0 installation and I'm using the Data Handler importer plugin via a MySQLi connection.
I have four entities declared:
<dataConfig>
<dataSource type="JdbcDataSource&...
0
votes
1answer
25 views
Concatenate values from different columns with Data Import Handler
I have a Django application which connects to a Solr instance for searching. How can I put the firstname and the surname in the same tune_author Solr field (see below)?
<entity name="tune&...
0
votes
0answers
17 views
Solr DataImportHandler cache with multiple keys
I'm trying to index data from a database using the Solr DataImportHandler. The database contains products defined by a class and with multiple features with the following relevant tables:
product ...
0
votes
0answers
14 views
Solr DIH: documents are not nested
I'm trying to import some data as nested documents.
I've tried to simplify my problem as much as possible.
Here my parent query:
SELECT 1 AS id,
'xxx' AS col1,
'yyy' AS col2
UNION
SELECT ...
0
votes
0answers
28 views
solr DIH: field composed by other fields
Here my sql query:
select id, 'AAA' AS one, 'BBB' AS two from tableX
DIH config:
<entity
name="item"
query="select id, 'AAA' AS one, 'BBB' AS two from tableX"
...
1
vote
1answer
59 views
solr DIH: RegExTransformer
Currently, I need to apply a transformation on bellow third column:
ACAC | 0 | 01
ACAC | 0 | 0101
ACAC | 0 | 0102
ACAC | 0 | 010201
I need to transform "010201" to "01/02/01".
So ...
0
votes
0answers
26 views
solr: tree-like structure DIH
I'm trying to denormalize a tree-like structure into solr.
I mean, into my relational database I've this:
+------------+
| ...
0
votes
0answers
12 views
solr DateFormatTransporter
I've set up and configured this DateFormatTransformer:
<field column="data_ini" name="data_inici_dt" dateTimeFormat="dd-MM-yyyy" locale="ca-ES"/>
At ...
0
votes
1answer
25 views
Solr SIH: Document is missing mandatory uniqueKey field: id
I'm getting this message:
Document is missing mandatory uniqueKey field: id
In order to fill this field on each document, I'm using:
<updateRequestProcessorChain name="files-update-processor&...
0
votes
0answers
32 views
Solr DIH: dataConfig is missing
I'm getting this message when I'm trying to perform my import handler:
Data Config problem: the root node <dataConfig> is missing
Here my data import handler:
<requestHandler name="/...
0
votes
1answer
20 views
solr DIH: Generate identity value
I need to import my database data into solr.
I'm using this data-config.xml:
<dataConfig>
<dataSource driver="org.hsqldb.jdbcDriver" url="jdbc:sqlserver://192.168.122.1;...
0
votes
0answers
18 views
SOLR Data-Import from SQL Server only fetches first few rows
I have SOLR 8.6 with DataImportHandler (Microsoft JDBC Driver for SQL Server 8.4) installed. I'm trying to import data calling stored procedure on SQL Server 2019. This procedure normally returns ...
0
votes
0answers
24 views
Solr DataImportHandler not indexing everything
i“m trying too update an old Solr instance, which is made for searching through Documents like Pdf“s, etc on a Fileserver. There are at least 100.000 files on it. I copied most of the Configs and ...
0
votes
0answers
48 views
Solr Dataimport nested entity
I wanna create nested entity with DataImportHandler.
I use Solr 8.6, Postgress 12, openjdk-11.
My config (schema.xml) looks like this:
<schema name="products" version="1.5">
...
2
votes
0answers
44 views
Solr Dataimport nested entity from PostgreSQL
I wanna create nested entity with DataImportHandler.
I use Solr 8.6, Postgress 12, openjdk-11.
My config (schema.xml) looks like this:
<schema name="products" version="1.5">
...
0
votes
0answers
38 views
SOLR nested fields produces messy result
I have a problem with SOLR.
The main motivation to do the thing I describe later was to remove the load from database into a SOLR. We are talking about an eshop which holds tons of products and where ...
1
vote
0answers
50 views
SOLR nested entities query (Oracle SQL DIH)
I want to setup my SOLR (8.5.2) schema in such a way that I can query a parent entity and get the child entities associated with it all in the same result. For example:
{
entityId: 1,
entityName: &...
1
vote
0answers
32 views
DataImport in SOLR
As I understand during data import into SOLR, all data is erased and a new one is added.
At the moment I have data that contains 4 fields. After importing from the SQL database(data contains 3 fields)...
1
vote
0answers
25 views
Logging Custom DataImportHandler Transformer in SOLR
I have created a custom DataImportHandler transformer for use in indexing contents from both the data base and the file system into SOLR. I can kick off the data import. I can see in SOLR's log (and ...
0
votes
1answer
47 views
Apache Solr : Data import handler exception - SetGraphicsStateParameters name for 'gs' operator not found in resources: /R7
configured data import handler to process bulk PDF documents. after process 21000 documents. Process going to idle and not processing all the documents.
When i see the log observed below things.
...
0
votes
1answer
49 views
SOLR 7 DataImportHandler ā Unable to get data into multivalued fields (via “Joined” table)
I'm attempting to use DIH to load our SOLR data.
I've done this on other SOLR cores/installations without issue, but for some reason I can't get it working on this installation.
The main data (media ...
0
votes
0answers
28 views
Indexing data from multiple data sources (CSV, RDBMS)
we are working on indexing data from multiple data sources using a single collection, for that specified data sources information in the data-config file and also updated managed-schema.xml by adding ...
0
votes
1answer
195 views
Connecting solr with aws RDS Mysql through data import handler
I recently started implementing solr-cloud on AWS EC2 for search applications. I have created 2 AWS Ec2 instances with the following configurations ---
EC2 Type - t2.medium
ram - 4GB
Disk Space - 8GB
...
0
votes
1answer
1k views
Solr java.lang.ClassNotFoundException: net.minidev.json.writer.JsonReaderI when executing DataImportHandler
I am currently working on a custom DataImportHandler for Solr 8.4.1.
I wrote my own entity processor class where I used the json-path library. And even though the solr-dataimporthandler (which I have ...
0
votes
0answers
12 views
Solr DataImport set field to specific value [duplicate]
I am trying to import documents from one collection to another using SolrEntityProcessor in /dataimport handler. I want set a field origin in the new document that will identify where the data came ...
1
vote
0answers
21 views
Can I pass a SQL result value to SolrEntityProcessor inside a single Entity?
I have a SQL table that stores Solr queries. I want to have a Solr collection that contains documents "stamped" with those Solr queries.
For example, I have a document in a RawDataCollection:
{
"...
0
votes
1answer
68 views
connect MongoDB and Solr 8.2 using DataImportHandler
I am trying to import mongoDB data for search using solr. I went through many documents but confused as I can see many solrconfig.xml file and one that present in _default (is the one I am using) even ...
0
votes
1answer
147 views
Solr Data Import Handler for Html
TLDR
How do I configure solr Data Import Handler so it will import html similar to solr's "post" utility ?
Context
We're doing a small project where code will export a set pages from wiki/...
0
votes
0answers
25 views
How to configure Solr Data Import Handler at run time?
I have integrated a Solr server with my business application as a full-text search provider. One core is configured to index from a file that is periodically uploaded to a secured location which is ...
2
votes
0answers
51 views
How to ignore copying of empty or null value from DB during Reindexing(DIH) to Solr
Is there any way to have a config in DIH which will ignore the null values while reindexing (DataImportHandler) from db to solr.
<field column="user_address " name="USR_Address"
template="ABC_${...
0
votes
1answer
184 views
How to pass customized parameters to SOlR DIH query
I have scenario where i need to pass customized parameters to solr data import query.
Ex- select * from customer where last_updated_date >=last_updated_indexed_date
The last_updated_indexed_date ...
0
votes
2answers
2k views
Uploading Data to Solr with Data import handler
Hi I am struggling to upload my data onto solr with the data import handler. What I am doing is starting a solr server using the command in the server folder
solr start
This then allows me to open ...
2
votes
0answers
55 views
Import Postgresql Array into Solr Field
I recently ran into a problem while trying to import Data from Postgres into solr:
In Postgres, i have an array with some values in it. I would like to import all values from those array into one ...
1
vote
1answer
212 views
Apache Solr still keep old data after delta import
I am using solr 7.6.
I do full import from mysql, the table customer looks like:
customer_id pk int
customer_code varchar
name varchar
update_datetime timestamp
I modify one ...
0
votes
0answers
1k views
Sql server data import - Error creating document : SolrInputDocument
I managed to set up a connection to my SQL server to import data into
Solr. The idea is to import filetables but for now I first want to get it
working using regular tables. So I created
data-...
1
vote
1answer
113 views
XML : Add new field with file URL
I have input file named myfile.xml with below content :
<?xml version="1.0" encoding="UTF-8"?>
<file-format>
<data-set xfer="1.2.840.10008.1.2.1" name="Little Endian Explicit">
<...
0
votes
1answer
50 views
SOLR LineEntityProcessor - fetched x records, but processed/indexed zero records
I am trying fetch all the hyperlinks from an html page and add them as documents to SOLR.
Here is my DIH config xml
<?xml version="1.0" encoding="UTF-8"?>
<dataConfig>
<...
0
votes
0answers
36 views
Solr-CustomTransformer For Converting Data from Database to table data
I am quite new to the solr custom transformer. I am trying to use solr custom customtransformer for preprocessing of the data fetched from Database.
The basic problem that, I am getting in using ...
0
votes
0answers
97 views
Solr: Modify the log for dih request
I am working on an application that uses solr as its search engine, I need to log the dih requests (with succeded & failed indexed documents).Is there a way to modify the log directory for a dih ...
0
votes
1answer
448 views
“ArrayList cannot be cast to java.lang.String” error message on Solr DIH endpoints
I am setting up a new Solr server and I'm running into an issue I haven't experienced in previous Solr installations. When I navigate to a core's "Dataimport" tab (without even triggering an import ...
1
vote
1answer
244 views
Onerror directives not working with Solr data import handler / PDFBox
we are trying to index about 10,000 PDFs against a MySQL table. We are using Solr 5.2.1, Tika 1.7 and PDFBox 1.8.8. Data import handler keeps giving various errors, which halts the entire process. ...
0
votes
1answer
184 views
Correct using ContentStreamDataSource in DIH
Solr documentation briefly mentions that ContentStreamDataSource can be used as a data source in the DIH config. No any other guidelines of hints are provided, so I expected that the following config ...
0
votes
1answer
135 views
Solr DIH slows when importing XML data from Oracle database
I'm working on a Solr DIH (DataImportHandler) task to import roughly 20 million documents stored in an Oracle database. Initially these imports will ramp up to over 500 docs a second, but within the ...
3
votes
2answers
637 views
Indexing entire XML document on SOLR 7 with no field specification
I would try to put an xml document on SOLR (now i'm using 7.3.0 version) without set specific fields in data-config or putting one tag to get all the others. I tried with schemaless mode but I didn't ...
1
vote
1answer
131 views
Apache Solr: How to make execution of a query in a sub-entity dependent on a field from the main entity
How can I make the execution of a query in a sub-entity dependent on a field
value from the main entity?
So as you can see in the (simplified) data config below, there's an entity
categories_lvl_0 ...
0
votes
1answer
164 views
How can I be notified in real-time when Solr is ready to accept another import request?
I'm writing a simple queue that sends update requests to a Solr core's DataImportHandler. This handler updates the Solr core by running a query against the database. When one update is complete, I'd ...
1
vote
0answers
509 views
Solr data import handler can't update but debug show it should
The problem is when we debug data import, it saids the values of a doc should change but then the value aren't on query.
This is the output of the dataimport.
{
"responseHeader": {
"status": 0,...
0
votes
1answer
315 views
ParentDeltaQuery not updating document in solr dataimporthandler
I am using solr's dataimporthandler to index postgres data in solr, my document structure has a parent entity and corresponding child entity: here is the data-config.xml file:
<entity name="...
0
votes
3answers
505 views
How to set Data Import Handler and Scheduler using solrJ Client
I am new to solr search, i have completed a simple search.
Now I want to index documents directly from Database and want set scheduler or trigger for updating index when there is any change in DB.
...