Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Logs UI] Add "Analyze in ML" buttons #48268

Merged
merged 104 commits into from
Oct 16, 2019
Merged
Show file tree
Hide file tree
Changes from 102 commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
4adc91d
Amend outer page / panel structure
Kerry350 Oct 3, 2019
62c7a6b
Remove unused imports
Kerry350 Oct 3, 2019
9793f4c
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
Kerry350 Oct 7, 2019
0cefd23
Add anomalies section and overall chart
Kerry350 Oct 7, 2019
b5f3bf8
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
Kerry350 Oct 8, 2019
7524569
Add legend
Kerry350 Oct 8, 2019
4119a05
Rename dataSet to partition
weltenwort Oct 8, 2019
8d887a8
Add max bucket anomaly score
weltenwort Oct 8, 2019
6ff0634
Move job and setup status types to common
weltenwort Sep 18, 2019
5eeefce
Move initial job status fetching into page content
weltenwort Sep 18, 2019
43c2edf
Poll for job status on results page
weltenwort Sep 18, 2019
bb157fd
Merge branch '47201-adapt-log-entry-rate-data-vis' of github.com:Kerr…
Kerry350 Oct 8, 2019
7811629
Add table / expanded row charts
Kerry350 Oct 8, 2019
ae059f0
Add bucket span text
Kerry350 Oct 8, 2019
a3820b7
Add stat section to expanded rows
Kerry350 Oct 8, 2019
1e8678d
Amend annotations on overall graph
Kerry350 Oct 8, 2019
427105d
Add rule to account for EuiFlexItem edge case
Kerry350 Oct 9, 2019
123084b
Move functions that handle derivations of data to a new file
Kerry350 Oct 9, 2019
737a25d
Tweak data points fetched
Kerry350 Oct 9, 2019
ded5117
Style bars in grey for anomalies charts
Kerry350 Oct 9, 2019
94fdb76
Add severity scoring to annotations
Kerry350 Oct 9, 2019
25d6e54
Fix default
Kerry350 Oct 9, 2019
7be35d2
Remove decimal places from anomaly score representations
Kerry350 Oct 9, 2019
a3d81da
Show all partitions and overall anomaly score in annotation tooltip f…
Kerry350 Oct 9, 2019
2f7855e
Handle 'unknown' to workaround lack of '' suuport in tables
Kerry350 Oct 10, 2019
78750b3
Add stats section to overall anomalies section
Kerry350 Oct 10, 2019
7e72fe4
Base x-domain off the series so that certain buckets aren't omitted
Kerry350 Oct 10, 2019
cb3d525
Tweak colours and DRY up annotation rendering
Kerry350 Oct 10, 2019
9db1834
Add sorting to table
Kerry350 Oct 10, 2019
a0908da
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
Kerry350 Oct 10, 2019
fd26252
Add "number of logs" to API results and render in UI stats
Kerry350 Oct 10, 2019
2569bc8
Track and render out-of-sync job configurations
weltenwort Oct 8, 2019
1813c80
Adjust translation labels
weltenwort Oct 11, 2019
ced21c8
Add stopped state callout
weltenwort Oct 11, 2019
992a7c5
Add more callout icons
weltenwort Oct 11, 2019
5591652
Merge branch 'master' into logs-ui-analysis-add-job-status-callouts
weltenwort Oct 11, 2019
87074d1
Fix api integration tests
weltenwort Oct 11, 2019
e453e07
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
Kerry350 Oct 14, 2019
3e181b0
Use "pretty" numbers for "Number of logs" stats
Kerry350 Oct 14, 2019
d65af63
Merge branch 'master' into logs-ui-analysis-add-job-status-callouts
weltenwort Oct 14, 2019
ce095aa
Improve status message wording
weltenwort Oct 14, 2019
c92615d
Change recreate job button color back to default
weltenwort Oct 14, 2019
bb01766
Add toolbar text
Kerry350 Oct 14, 2019
6100b51
Format all y axis values to 3 digits
Kerry350 Oct 14, 2019
36c7cb9
Remove "Overall anomaly score" and change all wording / calculations …
Kerry350 Oct 14, 2019
af7833c
Merge branch '47201-adapt-log-entry-rate-data-vis' of github.com:Kerr…
Kerry350 Oct 14, 2019
3031b69
Sort anomaly maximum scores for the overall chart tooltip
Kerry350 Oct 14, 2019
fbe41af
Remove unused translations
Kerry350 Oct 14, 2019
5f85146
Use white text with badge in toolbar
Kerry350 Oct 14, 2019
4df0469
Merge branch 'master' into logs-ui-analysis-add-job-status-callouts
weltenwort Oct 15, 2019
af01a97
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
Kerry350 Oct 15, 2019
f329127
Factor out a job recreation callout
weltenwort Oct 15, 2019
6e7f872
Replace `filter()[0]` with `find()` call
weltenwort Oct 15, 2019
40dbb59
Amend key
Kerry350 Oct 15, 2019
8e215d3
Use Math.round and introduce a formatAnomalyScore helper function
Kerry350 Oct 15, 2019
19ef762
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
32833e6
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
e732f5d
Format y-axis of log entry rate chart the same as anomalies charts
Kerry350 Oct 15, 2019
1548e14
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
b1a743a
Merge branch '47201-adapt-log-entry-rate-data-vis' of github.com:Kerr…
Kerry350 Oct 15, 2019
b76bf02
Remove grow prop
Kerry350 Oct 15, 2019
1cb4143
Update x-pack/legacy/plugins/infra/common/http_api/log_analysis/resul…
Kerry350 Oct 15, 2019
cadd37f
Update x-pack/legacy/plugins/infra/common/http_api/log_analysis/resul…
Kerry350 Oct 15, 2019
79c4e21
Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results…
Kerry350 Oct 15, 2019
63aca0b
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/page_re…
Kerry350 Oct 15, 2019
7bd5708
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
ee60082
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
157ddd8
Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results…
Kerry350 Oct 15, 2019
e4ed1d1
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
eb0b34a
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
a685a43
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
ad530c7
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
4c3f3f0
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
ad81285
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
0ce5bf6
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
1c587f1
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
8b7535c
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
b52d9c1
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
4bdfcfb
Update x-pack/legacy/plugins/infra/server/lib/log_analysis/log_analys…
Kerry350 Oct 15, 2019
a32f8e5
Update x-pack/legacy/plugins/infra/server/lib/log_analysis/log_analys…
Kerry350 Oct 15, 2019
8e31a6c
Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results…
Kerry350 Oct 15, 2019
721b7f8
Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results…
Kerry350 Oct 15, 2019
914790e
Merge branch '47201-adapt-log-entry-rate-data-vis' of github.com:Kerr…
Kerry350 Oct 15, 2019
a491bff
Change path
Kerry350 Oct 15, 2019
f7787bd
Amend property name
Kerry350 Oct 15, 2019
305cdf4
Use proper default value argument
Kerry350 Oct 15, 2019
7f468b4
Use Kibana dateFormat setting for toolbar formatting
Kerry350 Oct 15, 2019
7fa26ed
Change logic for calculating severity score categories
Kerry350 Oct 15, 2019
d01e5f1
Add missing translations
Kerry350 Oct 15, 2019
a1493ce
Add interface to top of file
Kerry350 Oct 15, 2019
8197183
Add no-wrap to tooltip
Kerry350 Oct 15, 2019
5bcad0d
Use more idomatic code
Kerry350 Oct 15, 2019
a5d6e45
Use static value for series styles
Kerry350 Oct 15, 2019
abae288
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
Kerry350 Oct 15, 2019
a688c3b
Merge branch '47201-adapt-log-entry-rate-data-vis' into logs-ui-analy…
weltenwort Oct 15, 2019
2727c69
Move the callouts into the right location
weltenwort Oct 15, 2019
ae7de65
Fix linter warning
weltenwort Oct 15, 2019
949ffa1
Add non-functional ML link button
weltenwort Oct 15, 2019
ab225d0
Add "Analyze in ML" buttons
Kerry350 Oct 15, 2019
8d72d24
Add "Analyze in ML" buttons (2)
Kerry350 Oct 15, 2019
1849578
Merge branch 'master' into 46445-add-ml-links
weltenwort Oct 16, 2019
1f530d1
Remove superfluous imports
weltenwort Oct 16, 2019
d483ed7
Tweak hierarchy of button and link to "Anomaly explorer" view for ove…
Kerry350 Oct 16, 2019
9dab86d
Use QueryString.encode instead of encodeURIComponent
Kerry350 Oct 16, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import createContainer from 'constate-latest';
import { useMemo, useCallback, useEffect } from 'react';

import { callGetMlModuleAPI } from './api/ml_get_module';
import { bucketSpan } from '../../../../common/log_analysis';
import { bucketSpan, getJobId } from '../../../../common/log_analysis';
import { useTrackedPromise } from '../../../utils/use_tracked_promise';
import { callJobsSummaryAPI } from './api/ml_get_jobs_summary_api';
import { callSetupMlModuleAPI, SetupMlModuleResponsePayload } from './api/ml_setup_module_api';
Expand Down Expand Up @@ -138,6 +138,12 @@ export const useLogAnalysisJobs = ({
fetchModuleDefinition();
}, [fetchModuleDefinition]);

const jobIds = useMemo(() => {
return {
'log-entry-rate': getJobId(spaceId, sourceId, 'log-entry-rate'),
};
}, [sourceId, spaceId]);

return {
fetchJobStatus,
isLoadingSetupStatus,
Expand All @@ -149,6 +155,7 @@ export const useLogAnalysisJobs = ({
viewSetupForReconfiguration,
viewSetupForUpdate,
viewResults,
jobIds,
};
};

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ export const AnalysisPageContent = () => {
const { sourceId, source } = useContext(Source.Context);
const { hasLogAnalysisCapabilites } = useContext(LogAnalysisCapabilities.Context);

const { setup, cleanupAndSetup, setupStatus, viewResults, fetchJobStatus } = useContext(
const { setup, cleanupAndSetup, setupStatus, viewResults, fetchJobStatus, jobIds } = useContext(
LogAnalysisJobs.Context
);

Expand All @@ -45,6 +45,7 @@ export const AnalysisPageContent = () => {
<AnalysisResultsContent
sourceId={sourceId}
isFirstUse={setupStatus === 'hiddenAfterSuccess'}
jobId={jobIds['log-entry-rate']}
/>
);
} else {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,10 @@ import numeral from '@elastic/numeral';
import { FormattedMessage } from '@kbn/i18n/react';
import moment from 'moment';
import React, { useCallback, useContext, useMemo, useState } from 'react';
import chrome from 'ui/chrome';
import euiStyled from '../../../../../../common/eui_styled_components';
import { TimeRange } from '../../../../common/http_api/shared/time_range';
import { bucketSpan } from '../../../../common/log_analysis';
import euiStyled from '../../../../../../common/eui_styled_components';
import { LoadingPage } from '../../../components/loading_page';
import {
LogAnalysisJobs,
Expand All @@ -35,20 +36,24 @@ import { useKibanaUiSetting } from '../../../utils/use_kibana_ui_setting';
import { FirstUseCallout } from './first_use';
import { AnomaliesResults } from './sections/anomalies';
import { LogRateResults } from './sections/log_rate';
import { getMlLinkFormatter } from './sections/helpers/ml_links';

const JOB_STATUS_POLLING_INTERVAL = 30000;

export const AnalysisResultsContent = ({
sourceId,
isFirstUse,
jobId,
Kerry350 marked this conversation as resolved.
Show resolved Hide resolved
}: {
sourceId: string;
isFirstUse: boolean;
jobId: string;
}) => {
useTrackPageview({ app: 'infra_logs', path: 'analysis_results' });
useTrackPageview({ app: 'infra_logs', path: 'analysis_results', delay: 15000 });

const [dateFormat] = useKibanaUiSetting('dateFormat', 'MMMM D, YYYY h:mm A');
const basePath = chrome.getBasePath();

const {
timeRange: selectedTimeRange,
Expand All @@ -60,7 +65,12 @@ export const AnalysisResultsContent = ({
const [queryTimeRange, setQueryTimeRange] = useState<TimeRange>(
stringToNumericTimeRange(selectedTimeRange)
);

const getMlLink = getMlLinkFormatter({
basePath,
jobId,
startTime: queryTimeRange.startTime,
endTime: queryTimeRange.endTime,
});
Kerry350 marked this conversation as resolved.
Show resolved Hide resolved
const bucketDuration = useMemo(() => {
// This function takes the current time range in ms,
// works out the bucket interval we'd need to always
Expand Down Expand Up @@ -214,6 +224,7 @@ export const AnalysisResultsContent = ({
setTimeRange={handleChartTimeRangeChange}
setupStatus={setupStatus}
timeRange={queryTimeRange}
getMlLink={getMlLink}
/>
</EuiPanel>
</EuiFlexItem>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/

import React from 'react';
import { EuiButton, ButtonColor } from '@elastic/eui';
import { FormattedMessage } from '@kbn/i18n/react';

export const AnalyzeInMlButton: React.FunctionComponent<{
color?: ButtonColor;
fill?: boolean;
href: string;
}> = ({ color = 'primary', fill = true, href }) => {
return (
<EuiButton color={color} fill={fill} href={href}>
<FormattedMessage
id="xpack.infra.logs.analysis.analyzeInMlButtonLabel"
defaultMessage="Analyze in ML"
/>
</EuiButton>
);
};
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import React, { useMemo } from 'react';
import { i18n } from '@kbn/i18n';
import numeral from '@elastic/numeral';
import { EuiFlexGroup, EuiFlexItem, EuiStat } from '@elastic/eui';
import { EuiFlexGroup, EuiFlexItem, EuiStat, EuiSpacer } from '@elastic/eui';
import { AnomaliesChart } from './chart';
import { GetLogEntryRateSuccessResponsePayload } from '../../../../../../common/http_api/log_analysis/results/log_entry_rate';
import { TimeRange } from '../../../../../../common/http_api/shared/time_range';
Expand All @@ -17,14 +17,17 @@ import {
formatAnomalyScore,
getTotalNumberOfLogEntriesForPartition,
} from '../helpers/data_formatters';
import { GetMlLink } from '../helpers/ml_links';
import { AnalyzeInMlButton } from '../analyze_in_ml_button';

export const AnomaliesTableExpandedRow: React.FunctionComponent<{
partitionId: string;
topAnomalyScore: number;
results: GetLogEntryRateSuccessResponsePayload['data'];
setTimeRange: (timeRange: TimeRange) => void;
timeRange: TimeRange;
}> = ({ results, timeRange, setTimeRange, topAnomalyScore, partitionId }) => {
getMlLink: GetMlLink;
}> = ({ results, timeRange, setTimeRange, topAnomalyScore, partitionId, getMlLink }) => {
const logEntryRateSeries = useMemo(
() =>
results && results.histogramBuckets
Expand Down Expand Up @@ -83,6 +86,12 @@ export const AnomaliesTableExpandedRow: React.FunctionComponent<{
)}
reverse
/>
<EuiSpacer size="s" />
<EuiFlexGroup>
<EuiFlexItem grow={false}>
<AnalyzeInMlButton fill={false} href={getMlLink(partitionId)} />
</EuiFlexItem>
</EuiFlexGroup>
</EuiFlexItem>
</EuiFlexGroup>
);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
*/

import {
EuiButton,
EuiEmptyPrompt,
EuiFlexGroup,
EuiFlexItem,
Expand All @@ -16,7 +15,6 @@ import {
} from '@elastic/eui';
import numeral from '@elastic/numeral';
import { i18n } from '@kbn/i18n';
import { FormattedMessage } from '@kbn/i18n/react';
import React, { useMemo } from 'react';

import euiStyled from '../../../../../../../../common/eui_styled_components';
Expand All @@ -29,9 +27,11 @@ import {
getLogEntryRateCombinedSeries,
getTopAnomalyScoreAcrossAllPartitions,
} from '../helpers/data_formatters';
import { GetMlLink } from '../helpers/ml_links';
import { AnomaliesChart } from './chart';
import { AnomaliesTable } from './table';
import { LogAnalysisJobProblemIndicator } from '../../../../../components/logging/log_analysis_job_status';
import { AnalyzeInMlButton } from '../analyze_in_ml_button';

export const AnomaliesResults: React.FunctionComponent<{
isLoading: boolean;
Expand All @@ -42,6 +42,7 @@ export const AnomaliesResults: React.FunctionComponent<{
timeRange: TimeRange;
viewSetupForReconfiguration: () => void;
viewSetupForUpdate: () => void;
getMlLink: GetMlLink;
}> = ({
isLoading,
jobStatus,
Expand All @@ -51,6 +52,7 @@ export const AnomaliesResults: React.FunctionComponent<{
timeRange,
viewSetupForReconfiguration,
viewSetupForUpdate,
getMlLink,
}) => {
const title = i18n.translate('xpack.infra.logs.analysis.anomaliesSectionTitle', {
defaultMessage: 'Anomalies',
Expand Down Expand Up @@ -105,12 +107,7 @@ export const AnomaliesResults: React.FunctionComponent<{
</EuiTitle>
</EuiFlexItem>
<EuiFlexItem grow={false}>
<EuiButton fill>
<FormattedMessage
id="xpack.infra.logs.analysis.analyzeInMlButtonLabel"
defaultMessage="Analyze in ML"
/>
</EuiButton>
<AnalyzeInMlButton href={getMlLink()} />
</EuiFlexItem>
</EuiFlexGroup>
<EuiSpacer size="m" />
Expand Down Expand Up @@ -193,7 +190,12 @@ export const AnomaliesResults: React.FunctionComponent<{
</EuiFlexItem>
</EuiFlexGroup>
<EuiSpacer size="l" />
<AnomaliesTable results={results} setTimeRange={setTimeRange} timeRange={timeRange} />
<AnomaliesTable
results={results}
setTimeRange={setTimeRange}
timeRange={timeRange}
getMlLink={getMlLink}
/>
</>
)}
</>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import { TimeRange } from '../../../../../../common/http_api/shared/time_range';
import { GetLogEntryRateSuccessResponsePayload } from '../../../../../../common/http_api/log_analysis/results/log_entry_rate';
import { AnomaliesTableExpandedRow } from './expanded_row';
import { getTopAnomalyScoresByPartition, formatAnomalyScore } from '../helpers/data_formatters';
import { GetMlLink } from '../helpers/ml_links';

interface TableItem {
id: string;
Expand Down Expand Up @@ -52,7 +53,8 @@ export const AnomaliesTable: React.FunctionComponent<{
results: GetLogEntryRateSuccessResponsePayload['data'];
setTimeRange: (timeRange: TimeRange) => void;
timeRange: TimeRange;
}> = ({ results, timeRange, setTimeRange }) => {
getMlLink: GetMlLink;
}> = ({ results, timeRange, setTimeRange, getMlLink }) => {
const tableItems: TableItem[] = useMemo(() => {
return Object.entries(getTopAnomalyScoresByPartition(results)).map(([key, value]) => {
return {
Expand Down Expand Up @@ -112,6 +114,7 @@ export const AnomaliesTable: React.FunctionComponent<{
topAnomalyScore={item.topAnomalyScore}
setTimeRange={setTimeRange}
timeRange={timeRange}
getMlLink={getMlLink}
/>
),
};
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/

import { encode } from 'rison-node';

export type GetMlLink = (partitionId?: string | undefined) => string;

export const getMlLinkFormatter = ({
basePath,
startTime,
endTime,
jobId,
}: {
basePath: string;
startTime: number;
endTime: number;
jobId: string;
}) => (partitionId?: string) => {
const startTimeParam = new Date(startTime).toISOString();
const endTimeParam = new Date(endTime).toISOString();

const _g = encode({
ml: {
jobIds: [jobId],
},
time: {
from: startTimeParam,
to: endTimeParam,
mode: 'absolute',
},
});

const baseLink = `${basePath}/app/ml#/timeseriesexplorer?_g=${_g}`;
Kerry350 marked this conversation as resolved.
Show resolved Hide resolved

if (partitionId) {
const _a = encode({
mlTimeSeriesExplorer: {
entities: { 'event.dataset': partitionId },
},
});
return `${baseLink}&_a=${encodeURIComponent(_a)}`;
} else {
return baseLink;
}
};
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,5 @@ import { npSetup } from 'ui/new_platform';
*/
export const useKibanaInjectedVar = (name: string, defaultValue?: unknown) => {
const injectedMetadata = npSetup.core.injectedMetadata;

return injectedMetadata.getInjectedVar(name, defaultValue);
};