3

I have a 64-bit Windows 7 machine with HANA Client installed and an ODBC connection to an SAP HANA database. In the ODBC Data Source administrator, I have added a connection to the required database and can successfully connect.

I am trying to use RStudio to retrieve data for analysis using R. I am finding that queries that return a handful of rows ("TOP 1" to "TOP 17") successfully return all 71 columns of data for the requested number of rows, but when I query with "TOP 18" or higher number of rows, I get all column titles, but 0 rows returned.

So the query:

res<-sqlQuery(ch, 'SELECT TOP 17 * FROM "SAPL2P"."/BIC/PZRPA_CNO" WHERE "/BIC/ZRPA_DCD"=\'CONFIRMED\'')

results in 17 rows of data, but

res2<-sqlQuery(ch, 'SELECT TOP 18 * FROM "SAPL2P"."/BIC/PZRPA_CNO" WHERE "/BIC/ZRPA_DCD"=\'CONFIRMED\'')

has 0 rows of data.

Any ideas what could be causing data not to be returned for more than 17 rows?

4
  • When running the queries in SAP HANA Studio, are all records returned?
    – Lars Br.
    Aug 16, 2016 at 8:48
  • I don't have HANA Studio installed. I might see if I can get it installed, but it should not be necessary. In the meantime, I'll ask a colleague to check if they get results back in HANA STUDIO.
    – db533
    Aug 16, 2016 at 13:54
  • From HANA STUDIO, the query does return the expected 18 records when run with TOP 18. TOP 200 also returned the top 200 records.
    – db533
    Aug 16, 2016 at 14:01
  • From the data extracted via HANA STUDIO, I do notice that in the 18th record, there is a character field that contains a Lithuanian character (an "E" with a dot above it). The first 17 rows do not have any such characters. Could this be a cause?
    – db533
    Aug 16, 2016 at 14:04

5 Answers 5

3

Ok, the problem here really is how R on Windows handles UTF data from ODBC (as already has been described). A quick search around SO shows that this problem is pretty common for R on Windows with a lot of different DBMS.

For SAP HANA what worked for me was to add the following parameter to the ODBC DSN (in the ODBC driver settings -> Settings... -> Special property settings):

CHAR_AS_UTF8 | TRUE

This makes SAP HANA ODBC to handle SQL_C_CHAR as UTF8.

2
  • 1
    Thanks, Lars. This change does enable the rows with accented characters to be returned, but the characters are converted incorrectly. Looks like there might be a further issue with codepages. Can this also be configured via the ODBC property settings, to use cp1257, for example?
    – db533
    Aug 17, 2016 at 5:32
  • 2
    OK, I found it! In addition to the above solution, I needed to indicate UTF-8 as the DBMS encoding when I opened the channel in RStudio: 'ch<-odbcConnect("HANA_QA_DS",uid="aaaaaaaa",pwd="bbbbbbbb", DBMSencoding="UTF-8")'.
    – db533
    Aug 17, 2016 at 10:10
0

The solution turned out to require 2 parts as the issue arose due to Baltic characters within the SAP data. The 2 steps were:

  • Adding the property 'CHAR_AS_UTF8' with a value of 'TRUE' in the ODBC DSN settings as Lars suggested above.
  • Opening the channel in RStudio with an extra parameter 'DBMSencoding="UTF-8"‌'​

So the RStudio command to open the channel to the HANA data is now: ch<-odbcConnect("HA‌​NA_QA_DS",uid="aaaaaa‌​aa",pwd="bbbbbbbb", DBMSencoding="UTF-8"‌​)

Thanks, Lars. Your input was instrumental to solving this!

0

I had the same issue. Adding the property 'CHAR_AS_UTF8' = 1 was not an option for me, since it has to be defined on the system of every new user using the code.

In my solution I cast the column to NCLOB.

res<-sqlQuery(ch, 'SELECT TO_NCLOB("COLUMNNAME") as "COLUMNNAME" FROM "SAPL2P"."MYSCHEMA"')

This might be complicated on the selection of *. However, mostly only certain known columns are affected by undesired characters.

0

I had a similar issue. At first, I had no rows returns at all. Adding believeNRows=FALSE and rows_at_time=1 to the initial connection helped in at least bringing back some data (until a failure occurred). In my case, I could only retrieve 5 rows with the 6th one failing which helped me to identify the problem row in the database.

In the end, I believe it has to do with encoding as per your suggestion above. I found that my issue was with the inverted single quote character.

I found the following information on the RODBC cran site:

The other internationalization issue is the character encoding used. When R and the DBMS are running on the same machine this is unlikely to be an issue, and in many cases the ODBC driver has some options to translate character sets. SQL is an ANSI (US) standard, and DBMSs tended to assume that character data was ASCII or perhaps currently this is stymied by bugs in the ODBC driver, so SQLColumns is unable to report on tables in specified databases. More recently DBMSs have started to (optionally or by default) to store data in Unicode, which unfortunately means UCS-2 on Windows and UTF-8 elsewhere. So cross OS solutions are not guaranteed to work, but most do.

SAP HANA is a database that store data in Unicode.

I tried various options to set the encoding:

  1. setting the DBMSencoding property in the RODBS.odbcConnect proc to UTF-8, latin1, ISO8859-1 and UCS-2 without any luck
  2. I also tried setting the encoding in RStudio itself without luck

In the end, I settled on creating a view in SAP HANA to replace the problem character in SQL. replace(content,x,y) as content where x was the problem character and y the replacement.

From that point on RODBC could retrieve the data without problem.

3
  • Looks like you had the 2nd half of the solution - adding the DBMSencoding property. :)
    – db533
    Aug 17, 2016 at 10:25
  • Glad it helped @db533. Unfortunately I still cannot get it to work on my side even after adding the CHAR_AS_UTF8 in my ODBC DSN. Could you perhaps tell me if you used TRUE or 1 as the value for the variable. In addition are you using the HDBODBC driver and which version? Thanks, Estee
    – Estee
    Aug 23, 2016 at 17:56
  • I used TRUE as the value for the variable. Yes, I am using the HDBODBC driver. Version is 1.00.112.04. Filename is LIBODBCHDB.DLL
    – db533
    Aug 25, 2016 at 13:26
0
$conn = odbc_connect("Driver=$driver;ServerNode=$host;Database=$db_name;CHAR_AS_UTF8=TRUE;", $username, $password,SQL_CUR_USE_DRIVER);

this is the sample connection how to give CHAR_AS_UTF8=TRUE in ODBC.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.