Support Environment Variable in conditionals #5115

githubmui · 2016-04-14T06:13:05Z

Hi,
in LS 2.3.0. I have got the problem that following expression doesn't evaluate to TRUE (if environment variable doesn't exists).

if "${LS_ENDP_JDBC:a}" == "a" { ... }

Thx in advance for reply.

suyograo · 2016-04-15T21:49:36Z

@githubmui The environment variable option was mostly added for substituting values, and not using it in conditional. But it seems this is a good usecase too. I'll keep this open as an enhancement.

githubmui · 2016-04-16T04:52:38Z

Hi,
its also an missing issue for the parameters of all the input plugins!
For example, it's not possible to use this nice feature for the URL of a "http_poller".

IrlJidel · 2016-04-25T12:03:06Z

As a workaround you could use mutate and metadata fields

mutate {
  add_field => { "[@metadata][LS_ENDP_JDBC]" => "${LS_ENDP_JDBC:a}" }
}

if [@metadata][LS_ENDP_JDBC] == "a" {
...
}

czerasz · 2016-07-16T11:31:05Z

The usage in if would make the configs more generic:

output {
  if "${OUTPUT:elasticsearch}" == "debug" {
    stdout { codec => rubydebug }
  } else {
    elasticsearch { ... }
  }
}

maya-sh · 2016-07-18T20:08:30Z

any update on this ? currently I am using the same approach as @IrlJidel, but it's more convenient to have the env. var. in the condition

berglh · 2016-12-08T00:51:39Z

I also want conditional inputs based on comparisons with environmental variables:

LS_DEV_MODE=false ./logstash --debug -e '
input { 
  if "${LS_DEV_MODE}" == "false" { 
    file { path => "/tmp/file1" sincedb_path => "/dev/null" start_position => "beginning" } 
  } 
  if "${LS_DEV_MODE}" == "true" { 
    file { path => "/tmp/file2" sincedb_path => "/dev/null" start_position => "beginning" } 
  }
}
output { 
  stdout { codec => "rubydebug" } 
}'

This is because I wish to have development and production inputs in the one configuration stack and control the mode of the pipeline using LS_DEV_MODE=(true|false).

maxlock · 2017-01-24T12:23:42Z

+1 I also would like to change behaviour based upon environment variables.

PhaedrusTheGreek · 2017-02-02T17:29:04Z

+1 for being able to use environment variables in conditionals.

astropuffin · 2017-02-09T01:20:58Z

Was very surprised and sad to find that I couldn't do this. I would have thought that environment variables would substitute anywhere in the config on load. Really confused as to why this isn't the case.

mrjameshamilton · 2017-02-09T10:27:20Z

+1

addykim · 2017-02-13T20:38:31Z

+1 although my desired use case is something like:

output {
    stdout { codec => rubydebug }
    if ${ELASTICSEARCH_HOST} is set {
        elasticsearch { 
            hosts => ["${ELASTICSEARCH_HOST}"]
            ...
        }
    }
}

Using psuedo code for the conditional because I'm not sure what the best way to check based on the current documentation, but that's a different topic.

berglh · 2017-02-13T23:05:00Z

You need to assign the environment variable to a field then do your if statement. I assign it to metadata fields so they don't make it through the output ie. `[@metadata][es-hosts]`

…

On 14 Feb. 2017 6:38 am, "Addy Kim" ***@***.***> wrote: +1 although my desired use case is something like: output { stdout { codec => rubydebug } if ${ELASTICSEARCH_HOST} is set { elasticsearch { hosts => ["${ELASTICSEARCH_HOST}"] ... } } } Using psuedo code for the conditional because I'm not sure what the best way to check based on the current documentation <https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html>, but that's a different topic. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5115 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AL4NqhRIchkUc_uDD7yja6f698g8cXG3ks5rcL9UgaJpZM4IHD4b> .

joniba · 2017-02-18T09:05:34Z

Can't get @IrlJidel 's workaround working on logstash 5.2.1. My conditionals are getting ignored. E.g., this always writes to localhost:9200:

input {
...
}

filter {
mutate {
add_field => { "[@metadata][ELASTIC_NOAUTH_URL]" => "${ELASTIC_NOAUTH_URL:x}" }
}
}

output {
if [@metadata][ELASTIC_NOAUTH_URL] != "x" {
elasticsearch {
hosts => [ "${ELASTIC_NOAUTH_URL:localhost:9200}" ]
}
}
}

I've tried miriad variations to no success :(

danizen · 2017-06-08T20:29:13Z

+1 - just found this, and tried the other method first.

nugend · 2017-08-04T20:32:47Z

Can elastic please call this out in documentation or put a warning for it at least? Got blocked for most of a day wondering what the hell I was doing wrong.

mihomir · 2017-09-05T14:33:20Z

+1
I've been debugging a Logstash configuration and the condition that should be used to identify the header in a CSV file is supposed to you environmental variables
if ["${COLUMN_NAME_USED_TO_SKIP_HEADER_ROW}"] == "${COLUMN_NAME_USED_TO_SKIP_HEADER_ROW}" {
And it appears that it is not working because of this issue.
Which is weird, when other parts of the Logstash configuration also use those variables and work as expected.

berglh · 2018-08-09T03:23:38Z

Would be great to see this feature being worked on.

duaraghav8 · 2019-07-13T10:59:11Z

Hi,
Has there been any progress on this issue?
Although it isn't a blocker for me, my workaround is pretty ugly and having this feature would make my pipeline configurations much more sane :)

dormunis · 2019-10-02T09:13:19Z

any progress?

shubydo · 2020-01-28T21:28:28Z

Any update on this issue? This is something we would really like to use in our pipelines.

iTaybb · 2020-03-16T09:46:33Z

Can we please push this? It's a very useful feature that been requested for years.

MorrieAtElastic · 2020-04-24T08:18:04Z

I have a customer who opened a Support Ticket because his use of conditional logic on an env variable was being rejected as an error.

In fact this is a very desirable enhancement and the existing documentation, while not showing examples of using an ENV within a conditional statement, does not explicitly forbid such use.

ceeeekay · 2020-07-21T09:05:13Z

This is affecting me too. I just spent hours backtracking and making changes to my configs on a new build that's using a lot of variables.

Any progress on this?

brunocascio · 2020-09-09T21:00:01Z

Anyone using env variables with conditionals?

I need something like this:

output {
  if "${IS_LOCAL}" == "true" {
    stdout {}
  } else {
    gelf {}
  }
}

asmith-elastic · 2021-04-26T23:23:32Z

There have been a few users that have expressed interest in this functionality, as this would be a desirable enhancement in order to satisfy their use cases.

+1 for being able to use environment variables in conditionals.

Alexandre-Genon · 2021-05-19T15:27:03Z

I'm also voting for this one, it will definitely be helpful to control the Logstash pipelines using environment variables.

ay27 · 2021-06-11T03:41:31Z

any progress?

makefu · 2021-09-10T12:46:20Z

I'd love to see this feature as well!

Environment variable expansion only works in plugin parameters, not in conditionals. For more on this limitation see #5115

) Environment variable expansion only works in plugin parameters, not in conditionals. For more on this limitation see elastic#5115 (cherry picked from commit 096eb7a)

…13311) Environment variable expansion only works in plugin parameters, not in conditionals. For more on this limitation see #5115 (cherry picked from commit 096eb7a)

…13312) Environment variable expansion only works in plugin parameters, not in conditionals. For more on this limitation see #5115 (cherry picked from commit 096eb7a)

This PR substitutes ${VAR} in Expression, except RegexValueExpression, with the value in secret store, env. The substitution happens after syntax parsing and before graph execution. Fixed: #5115

This PR substitutes ${VAR} in Expression, except RegexValueExpression, with the value in secret store, env. The substitution happens after syntax parsing and before graph execution. Fixed: elastic#5115

This PR substitutes ${VAR} in Expression, except RegexValueExpression, with the value in secret store, env. The substitution happens after syntax parsing and before graph execution. Fixed: #5115

kaisecheng · 2022-01-26T11:46:10Z

Logstash 7.17 has a fix for this issue

brunobastosg · 2022-05-24T16:38:41Z

Is it possible to use this feature to check if an environment variable is set?

Something like:

input {
  if "${BASE_URL}" {
    http_poller {
      urls => {
        url => "${BASE_URL}/api/v1"
      }
      request_timeout => 120
      socket_timeout => 60
      schedule => { cron => "0 7 * * *" }
      codec => "json"
    }
  }
}

I'm having trouble getting this to work.

kaisecheng · 2022-05-24T17:38:07Z

@brunobastosg You can alternatively check by the following

if ("${VAR:}" != "") {
    http_poller { ... }
 }

You can give a default value by using the form ${VAR:default_value}. Logstash uses the default value if the environment variable is undefined.

brunobastosg · 2022-05-24T17:48:46Z

@kaisecheng does logstash still try to parse the http_poller block? Because after I tried your suggestion I got the following error:

[2022-05-24T17:42:23,751][INFO ][logstash.inputs.http_poller][main] Registering http_poller Input {:type=>"myType", :schedule=>{"cron"=>"* * * * *"}, :timeout=>nil}
[2022-05-24T17:42:24,151][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Invalid URL /api/v1>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.1.0/lib/logstash/inputs/http_poller.rb:142:in `validate_request!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.1.0/lib/logstash/inputs/http_poller.rb:134:in `normalize_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.1.0/lib/logstash/inputs/http_poller.rb:65:in `block in setup_requests!'", "org/jruby/RubyHash.java:1415:in `each'", "org/jruby/RubyEnumerable.java:886:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.1.0/lib/logstash/inputs/http_poller.rb:65:in `setup_requests!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.1.0/lib/logstash/inputs/http_poller.rb:55:in `register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:390:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:315:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x2f63074d run>"}

Apparently it tried to validate the url ${BASE_URL:}/api/v1, but since BASE_URL is undefined, the url becomes /api/v1 which is not valid.

Here is the logstash.conf file:

input {
 # the http input is always available
  http {
    host => "0.0.0.0"
    port => 8080
  }
 # the http_poller input should only be available if BASE_URL is set
  if ("${BASE_URL:}" != "") {
    http_poller {
      urls => {
        url => "${BASE_URL:}/api/v1"
      }
      request_timeout => 120
      socket_timeout => 60
      schedule => { cron => "* * * * *" }
      codec => "json"
      type => "myType"
    }
  }
}

output {
  stdout { codec => rubydebug }
}

kaisecheng · 2022-05-24T18:23:30Z

@brunobastosg if statement only works in filter { } and output { }

brunobastosg · 2022-05-24T18:38:56Z

@brunobastosg if statement only works in filter { } and output { }

Sorry, I hadn't noticed that in the documentation. I assumed it could be anywhere.

Thank you for your time!

thom-vend · 2022-06-08T05:05:13Z

FYI . VAR cannot be evaluated to a number, hence cannot do number comparison.

suyograo added bug enhancement and removed bug labels Apr 15, 2016

suyograo changed the title ~~Problem with Environment Variable (LS 2.3.0)?~~ Support Environment Variable in conditionals Apr 15, 2016

filex mentioned this issue Oct 18, 2016

Conditions in input section are ignored #6080

Open

wibbly13 mentioned this issue Aug 4, 2020

Access control for browsers stops working when basic auth is enabled logstash-plugins/logstash-input-http#83

Open

roaksoax added the int-shortlist label Oct 6, 2021

jsvd added a commit that referenced this issue Oct 7, 2021

[DOC] Clarify the scope of environment variable expansion

251f19a

Environment variable expansion only works in plugin parameters, not in conditionals. For more on this limitation see #5115

jsvd mentioned this issue Oct 7, 2021

[DOC] Clarify the scope of environment variable expansion #13299

Merged

jsvd added a commit that referenced this issue Oct 13, 2021

[DOC] Clarify the scope of environment variable expansion (#13299)

096eb7a

Environment variable expansion only works in plugin parameters, not in conditionals. For more on this limitation see #5115

jsvd mentioned this issue Oct 13, 2021

Backport PR #13299 to 7.x: [DOC] Clarify the scope of environment variable expansion #13311

Merged

jsvd mentioned this issue Oct 13, 2021

Backport PR #13299 to 7.15: [DOC] Clarify the scope of environment variable expansion #13312

Merged

kaisecheng mentioned this issue Nov 1, 2021

support environment variable in if statement #13370

Closed

5 tasks

andsel assigned kaisecheng Dec 16, 2021

kaisecheng mentioned this issue Jan 17, 2022

Support env variable in condition #13608

Merged

5 tasks

kaisecheng closed this as completed in #13608 Jan 25, 2022

Support Environment Variable in conditionals #5115

Support Environment Variable in conditionals #5115

Comments

githubmui commented Apr 14, 2016

suyograo commented Apr 15, 2016

githubmui commented Apr 16, 2016

IrlJidel commented Apr 25, 2016 • edited Loading

czerasz commented Jul 16, 2016

maya-sh commented Jul 18, 2016

berglh commented Dec 8, 2016

maxlock commented Jan 24, 2017

PhaedrusTheGreek commented Feb 2, 2017

astropuffin commented Feb 9, 2017

mrjameshamilton commented Feb 9, 2017

addykim commented Feb 13, 2017

berglh commented Feb 13, 2017 via email

joniba commented Feb 18, 2017 • edited Loading

danizen commented Jun 8, 2017

nugend commented Aug 4, 2017

mihomir commented Sep 5, 2017

berglh commented Aug 9, 2018

duaraghav8 commented Jul 13, 2019

dormunis commented Oct 2, 2019

shubydo commented Jan 28, 2020

iTaybb commented Mar 16, 2020 • edited Loading

MorrieAtElastic commented Apr 24, 2020

ceeeekay commented Jul 21, 2020

brunocascio commented Sep 9, 2020 • edited Loading

asmith-elastic commented Apr 26, 2021

Alexandre-Genon commented May 19, 2021

ay27 commented Jun 11, 2021

makefu commented Sep 10, 2021

kaisecheng commented Jan 26, 2022

brunobastosg commented May 24, 2022 • edited Loading

kaisecheng commented May 24, 2022

brunobastosg commented May 24, 2022 • edited Loading

kaisecheng commented May 24, 2022

brunobastosg commented May 24, 2022

thom-vend commented Jun 8, 2022

IrlJidel commented Apr 25, 2016 •

edited

Loading

joniba commented Feb 18, 2017 •

edited

Loading

iTaybb commented Mar 16, 2020 •

edited

Loading

brunocascio commented Sep 9, 2020 •

edited

Loading

brunobastosg commented May 24, 2022 •

edited

Loading

brunobastosg commented May 24, 2022 •

edited

Loading