8

I am using FSharp.Data HTMLProvider to extract table rows:

let [<Literal>] URL = "../DailyResultsType.html"
type RawResults = HtmlProvider<URL>
let results = RawResults.Load(URL).Tables
let dailySeq =
    results.Table2.Rows
    |> Seq.tail

Looping over the rows (row is a seq<HtmlProvider<...>.Table2.Row>):

for row in dailySeq do
    printfn "%A" row

Results in:

(1, nan, nan, 2)
(1, nan, nan, 3)
~~~

Columns like #2 and #3 are getting auto-typed by the provider as decimal and double as the HTML contain strings such as "$12.00" or "$12".

  • Can I dynamically change the type of those columns within the type returned from the HtmlProvider<URL> at runtime (i.e. from double to string) (but I would prefer a numeric type so I can Deedle the results)

  • Or apply a runtime string transformation to the values in those columns to remove the non-digit characters so they are valid decimal/double/int types...

  • Or am I missing a basic concept (most likely as I am a F# noobie)

4
  • 1
    Can you please share a small HTML file that reproduces the problem? Aug 10, 2016 at 15:17
  • does it return nan on all values? the typeprovider will replace with nan some missing values. You can filter with Double.isNan. Decimal and double are numeric types... You can use string to convert it. Best would be an example html in gist or a link.
    – s952163
    Aug 11, 2016 at 1:50
  • @s952163 Yes, they are all nan (500+ rows) * 60+ pages . I'll try to create a publishable sample page that reproduces the issue... So far it is easier just to pre-parse the pages and then feed them to the HTMLProvider Aug 11, 2016 at 9:25
  • I think you will get all nan when the column contains both some monetary value and N/As.
    – s952163
    Aug 11, 2016 at 10:18

1 Answer 1

0

I think you should try to set PreferOptionals to true, that way the N/As will become null, and the rest of the number decimals.

type HtmlType = HtmlProvider<URL,PreferOptionals=true> or
type HtmlType = HtmlProvider<URL,PreferOptionals=true,Culture="en-US">

let results = HtmlType.Load(URL)
results.Tables.Table1.Rows
// val it : HtmlProvider<...>.Table1.Row [] =
// [|("Jill", "Smith", Some 50.0M); ("Eve", "Jackson", Some 100000M);
// ("John", "Doe", Some 100M); ("Jane", "Doe", null)|]

If there is no missing value in the table, and I omit PreferOptionals and Culture then I get the following output:

//val it : HtmlProvider<...>.Table1.Row [] =
//[|("Jill", "Smith", 50.0M); ("Eve", "Jackson", 100000M);
//("John", "Doe", 100M)|]

By the way, I might be wrong but I couldn't find anything where you can specify the schema of the table like in the csv provider. So once I get the data out I would just directly work with the tuple array, if there are not that many elements that should be straightforward. You can use string if necessary or pipe it directly to Deedle (rows |> Frame.ofRecords).

I used the following sample table.

<table style="width:100%">
  <tr>
    <th>Firstname</th>
    <th>Lastname</th> 
    <th>Age</th>
  </tr>
  <tr>
    <td>Jill</td>
    <td>Smith</td> 
    <td>$50.0</td>
  </tr>
  <tr>
    <td>Eve</td>
    <td>Jackson</td> 
    <td>$100,000</td>
  </tr>
  <tr>
    <td>John</td>
    <td>Doe</td> 
    <td>$100</td>
  </tr>
  <tr>
    <td>Jane</td>
    <td>Doe</td> 
    <td>N/A</td>
  </tr>
</table> 

2
  • Thanks, but there are no N/A's in those columns, all the values are prefix by $ symbol and are thousand comma formatted with and without decimal (cents) value. I've manually reviewed each table row to confirm. Aug 11, 2016 at 11:11
  • In that case some sort of sample would be helpful. But either way I think we established that parsing the html table is an iffy endevaour with the type provider so you might be better off doing it outside. I will see if mixing decimal and non-decimal will make any difference. However using a $ symbol should give you a decimal value (M). If not than it's a locale issue. Is there any way you could set locale to US for example in the type provider? The parameter is Culture.
    – s952163
    Aug 11, 2016 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.