Data Reader to Cube, can't find log file

Options
Matthew Brown
Matthew Brown Customer
Name Dropper Level 200: Leveraging Board for Business Insights Level 100: Foundations of Building in Board First Comment
edited January 2023 in Platform
another newbie question, thanks in advance for anyone that can assist !
  • I have several dimensions / entities + a flat file for a cube, in CSV file on AWS S3
  • I am able to load the dimensions with no issue
  • I load the csv cube file, check the 'log' and 'stop on error' boxes
  • My log file is not in the same S3 bucket with "_LOG" 
  • 221k files are valid, 594k are discarded - I'm trying to figure out why those won't load - also, the cube seems completely empty when I try to create a basic report on the 221k files that it says were not discarded
  • am i missing some obvious config?
    image
------------------------------
Matt Brown
Manager, Business Intelligence
Ebsco Industries Inc.
United States
------------------------------

Answers

  • Stephane Joly
    Stephane Joly Active Partner
    First Anniversary First Comment 5 Up Votes
    edited January 2023
    Options
    Hi Matt,
    It could be an issue with a special character and especially if in one of the lines (probably 221879) there a line return character. It could also be an issu with the format of a field, for instance if you have a non numeric character in a numeric field (event a "space" , "-", etc.)

    Hope this helps.

    Stéphane Joly

    ------------------------------
    Stephane Joly
    Manager
    CGI France
    France
    ------------------------------
    -------------------------------------------
  • Matthew Brown
    Matthew Brown Customer
    Name Dropper Level 200: Leveraging Board for Business Insights Level 100: Foundations of Building in Board First Comment
    edited January 2023
    Options
    Thank you for the response, Stephane.

    Great input, I'll check it.

    Any ideas around why the log file might not be there in S3 AWS bucket?

    And how does the "Stop on error" work?  Similar to a debugger?

    ------------------------------
    Matt Brown
    Manager, Business Intelligence
    Ebsco Industries Inc.
    United States
    ------------------------------
    -------------------------------------------
  • Stephane Joly
    Stephane Joly Active Partner
    First Anniversary First Comment 5 Up Votes
    edited January 2023
    Options
    My understanding is it just stops,meaning it should import data until their is an error and then stops reading the file. Therefore in this case I imagine their is not log or only for the first line.

    ------------------------------
    Stephane Joly
    Manager
    CGI France
    France
    ------------------------------
    -------------------------------------------
  • Nico Weber
    Nico Weber Active Partner
    May Badge of the Month New Community Member April Badge of the Month March Badge of the Month
    edited February 2023
    Options

    Hello Matt,

    In the bug fix notes for the latest Board 2023 Spring Release, I found an entry that I think fits perfectly with the problem you describe.

    The bug fix note is as follows:

    • Data reader. In the case of a Text file Data reader, the log file with all discarded records was not created in the same path as the source text file if the latter was on a supported cloud storage service (Amazon S3, Azure Blob Storage, Google Cloud Storage). The issue has been fixed and a new behavior has been put in place: the log file is now temporarily hosted in the default storage path in the Board cloud environment and Board will try to upload it on the cloud storage path where the source text file resides.
      • If this process is successful, the temporary file in the Board cloud environment will be deleted. 
      • If this process fails, then the log file is maintained in the default storage path in the Board cloud environment
    So an upgrade to the 2023 Spring Release may solve this problem.

    Kind regards
    Nico

    ------------------------------
    Nico Weber
    Senior Associate
    PricewaterhouseCoopers GmbH Wirtschaftsprüfungsgesellschaft
    Germany
    ------------------------------
    -------------------------------------------

Categories