9

If I'd like to run a Flink app locally, directly from within Intellij but I need to specify config params (like fs.hdfs.hdfssite to set up S3 access), is there any other way to provide those config params apart from ExecutionEnvironment.createLocalEnvironment(conf)? What if I want to use StreamExecutionEnvironment.getExecutionEnvironment? Can I have a Flink config in my project and point the local app to it?

Is this the proper way to do it? Or would you set up your IDE to submit the app to a real local Flink instance?

4 Answers 4

4

To create a StreamExecutionEnvironment with configuration options, use this call StreamExecutionEnvironment.createLocalEnvironment(int parallelism, Configuration configuration)

4

I could load the flink-conf.yaml file by providing the ENV_VAR FLINK_CONF_DIR in IntelliJ Run Configuration

2

You can use StreamExecutionEnvironment.getExecutionEnvironment() -- it will figure out the current setting and if executed in an IDE will return an "LocalEnvironemnt".

Thus, should be able to use conf/flink-conf.yaml to set your configuration values.

6
  • So I could put a flink-conf.yaml in my projects /src/main/resources/conf and the execution environment would find it?
    – bibac
    Commented Jun 10, 2016 at 12:19
  • Not sure... I guess conf/flink-conf.yaml should be in your IDE working directory. Commented Jun 10, 2016 at 13:25
  • 5
    @bibac @matthias-j-sax Can you tell me where I should put conf/flink-conf.yaml while running Flink job from IDEA?
    – Grzes
    Commented Oct 4, 2017 at 12:33
  • 2
    The local environment does not appear to know how to load a config file. I had to manually create a Configuration from ConfigurationUtils.createConfiguration and a Properties. The environment I then created with StreamExecutionEnvironment.createLocalEnvironment(n, conf)
    – Avi Cherry
    Commented Jul 31, 2019 at 20:18
  • 1
    I have used createRemoteEnvironment(host, port, null) while my flink is running locally. Commented Sep 4, 2019 at 9:24
0

Add the following dependency to your pom.xml:

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-metrics-jmx_2.11</artifactId>
    <version>${flink.version}</version>
</dependency>

Then change your job to use the following StreamExecutionEnvironment:

Properties props = new Properties();
props.put("metrics.reporter.jmx.factory.class", "org.apache.flink.metrics.jmx.JMXReporterFactory");
Configuration conf = ConfigurationUtils.createConfiguration(props);
StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);

If you prefer to use createLocalEnvironment, it works in the same way but without the Flink dashboard.

In this blog post you can find a demo, including source code and screenshots.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.