Commit 466ba3a0 authored by Schoumi's avatar Schoumi

Connection, retrieve list of past declaration, fetch each one done

parents
# editorconfig.org
# editorconfig is a unified configuration file that all editors can take
# into account
root = true
[*]
charset = utf-8
end_of_line = lf
trim_trailing_whitespace = true
insert_final_newline = true
indent_style = space
indent_size = 2
# Connector config
.account
.token.json
konnector-dev-config.json*
fixtures/*
data/*
# NPM
node_modules/
npm-debug.log
yarn-error.log
# Build
build/
# OS generated files
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db
desktop.ini
# Editors / IDEs
.floo
.flooignore
.idea/
*.swp
*.swo
*.tmp
# Default
# /!\ KEEP THIS SECTION THE LAST ONE
!.gitkeep
*.pdf
build
node_modules
package.json
language: node_js
node_js:
- '8'
env:
global:
- GH_USER_EMAIL="travis@example.org"
- GH_USER_NAME="cozy-bot"
- DEPLOY_REPOSITORY="git@github.com:konnectors/cozy-konnector-template.git"
# Token to access Cozy Registry, you should replace it with a token specific to this konnector
- secure: CQ44ZVjPAIVRvGgMcduSAX4VmaOrzF3nrJXPFUsNQ2tMVVdaAlI1zTNkdfntqigZgz6Mpk9x+2FcA8qrkNzQ4cW0lPP9f5dBtSH1xetfpXDFKprW4bqzmhgXSXJFTOYxkChOBPmCYJr+Vv4XqDkv/dBEPdfRSgvVyd5YxHSd5HBU91rKyR+Hox7LTnhmlXleLIs11EQgp7RK+L28MBCtCqAa2niFlKKRlqupm2FSpZzxCh9qeY0wKCQzPE+QZUm4waJ1SUGKQwbzAfyOxu2O6rwzaTSknfdJoSX6AMxjRTWBC9c3VfUGM3wjVJIqMhiUj2FQhVIjiDyX/W60/q3UMC5WYzSvEDDfhnKCPwiUKSU7HvSxx9q2YP34evxKo7KPahz9zLlQZvW+GBMyI9hbMDBTALuQVVzhftsvqfrogyYhkEPviiYiVADWNxhacsbUjRA5svXnZvtmCPo7fLFwel/krgC5Z8AwAY0YxWjSHNVScUkieSDCrlxrC09vTtG60unt9gtXbD5ruUkDLWSUwHBv2p0WY1TdP+2Vlf3yYfBvEVbqi59AxG+T17bq7Xv3QZnkrwC+UOCNCeg+gHPv+fRpUaYWzCO4uaoY/nlJa7RulQFhGca3A//6xRo/KgWw9JJnOXJelNjE9oHCUu9xShotUZKOF+O93V/BRqs9ITo=
cache:
yarn: true
directories:
- node_modules
branches:
except:
- build
- build-debug
script:
- yarn lint
- yarn build
- yarn run check
deploy:
- provider: script
skip-cleanup: true
script: DEPLOY_BRANCH=build yarn deploy && yarn cozyPublish
on:
branch: master
- provider: script
skip-cleanup: true
script: DEPLOY_BRANCH=build yarn deploy && yarn cozyPublish
on:
tags: true
before_install:
# run the yarn travisDeployKey command to genrate this
# - openssl aes-256-cbc -K $encrypted_0b21ecdcf920_key -iv $encrypted_0b21ecdcf920_iv
# -in github_deploy_key.enc -out /tmp/github_deploy_key -d
- eval "$(ssh-agent -s)"
- chmod 600 /tmp/github_deploy_key
- ssh-add /tmp/github_deploy_key
after_deploy:
- rm /tmp/github_deploy_key
- ssh-add -D
# General code owners
* @konnectors/maintainers
How to contribute to Cozy Connector?
====================================
Thank you for your interest in contributing to Cozy! There are many ways to contribute, and we appreciate all of them.
Security Issues
---------------
If you discover a security issue, please bring it to our attention right away! Please **DO NOT** file a public issue, instead send your report privately to security AT cozycloud DOT cc.
Security reports are greatly appreciated and we will publicly thank you for it. We currently do not offer a paid security bounty program, but are not ruling it out in the future.
Bug Reports
-----------
While bugs are unfortunate, they're a reality in software. We can't fix what we don't know about, so please report liberally. If you're not sure if something is a bug or not, feel free to file a bug anyway.
Opening an issue is as easy as following [this link][issues] and filling out the fields. Here are some things you can write about your bug:
- A short summary
- What did you try, step by step?
- What did you expect?
- What did happen instead?
- What is the version of the Cozy Connector?
Pull Requests
-------------
Please keep in mind that:
- Pull-Requests point to the `master` branch
- You need to cover your code and feature by tests
- You may add documentation in the `/docs` directory to explain your choices if needed
- We recommend to use [task lists][checkbox] to explain steps / features in your Pull-Request description
- You do __not__ need to build app to submit a PR
### Workflow
Pull requests are the primary mechanism we use to change Cozy. GitHub itself has some [great documentation][pr] on using the Pull Request feature. We use the _fork and pull_ model described there.
- [ ] clone the repository with `git clone`
- [ ] add a remote to your profile with `git remote add fork`
- [ ] create a new branch for the work with `git checkout --branch my-branch origin/master`
- [ ] add some changes with `git add --patch .`
- [ ] create at least one commit with `git commit --message '{fix,feat,chore,test,doc}: a 70 character long description of the changes'`
- [ ] check styleguide, formating, linting and tests
- [ ] push to your remote with `git push fork my-branch`
- [ ] create a Pull-Request with a detailed description
Writing documentation
---------------------
Documentation improvements are very welcome. We try to keep a good documentation in the `/docs` folder. But, you know, we are developers, we can forget to document important stuff that look obvious to us. And documentation can always be improved.
Community
---------
You can help us by making our community even more vibrant. For example, you can write a blog post, take some videos, answer the questions on [the forum][forum], organize new meetups, and speak about what you like in Cozy!
[issues]: ../../issues
[pr]: https://help.github.com/categories/collaborating-with-issues-and-pull-requests/
[forum]: https://forum.cozy.io/
This diff is collapsed.
[Cozy][cozy] <YOUR SUPER NEW KONNECTOR NAME>
=======================================
What's Cozy?
------------
![Cozy Logo](https://cdn.rawgit.com/cozy/cozy-guidelines/master/templates/cozy_logo_small.svg)
[Cozy] is a personal data platform that brings all your web services in the same private space. With it, your webapps and your devices can share data easily, providing you with a new experience. You can install Cozy on your own hardware where no one's tracking you.
What is this konnector about ?
------------------------------
This konnector retrieves your <SOME DATA> and <SOME OTHER DATA> from <SERVICE>
<YOUR DESCRIPTION HERE>
### Open a Pull-Request
If you want to work on this konnector and submit code modifications, feel free to open pull-requests!
</br>See :
* the [contributing guide][contribute] for more information about how to properly open pull-requests.
* the [konnectors development guide](https://docs.cozy.io/en/dev/konnector/)
### Run and test
Create a `konnector-dev-config.json` file at the root with your test credentials :
```javascript
{
"COZY_URL": "http://cozy.tools:8080",
"fields": {"login":"zuck.m@rk.fb", "password":"123456"}
}
```
Then :
```sh
yarn
yarn standalone
```
For running the konnector connected to a Cozy server and more details see [konnectors documentation](https://docs.cozy.io/en/dev/konnector/)
### Cozy-konnector-libs
This connector uses [cozy-konnector-libs](https://github.com/cozy/cozy-konnector-libs). It brings a bunch of helpers to interact with the Cozy server and to fetch data from an online service.
### Maintainer
The lead maintainers for this konnector is <YOUR NAME>
### Get in touch
You can reach the Cozy Community by:
- [konnectors documentation](https://docs.cozy.io/en/dev/konnector/)
- Chatting with us on IRC [#cozycloud on Freenode][freenode]
- Posting on our [Forum]
- Posting issues on the [Github repos][github]
- Say Hi! on [Twitter]
License
-------
<YOUR KONNECTOR NAME> is developed by <your name> and distributed under the [AGPL v3 license][agpl-3.0].
[cozy]: https://cozy.io "Cozy Cloud"
[agpl-3.0]: https://www.gnu.org/licenses/agpl-3.0.html
[freenode]: http://webchat.freenode.net/?randomnick=1&channels=%23cozycloud&uio=d4
[forum]: https://forum.cozy.io/
[github]: https://github.com/cozy/
[nodejs]: https://nodejs.org/
[standard]: https://standardjs.com
[twitter]: https://twitter.com/mycozycloud
[webpack]: https://webpack.js.org
[yarn]: https://yarnpkg.com
[travis]: https://travis-ci.org
[contribute]: CONTRIBUTING.md
#!/bin/bash
set -e
if [ -z "$GITHUB_TOKEN" ]; then
echo ""
echo "GITHUB_TOKEN environment variable is missing"
echo ""
echo "TRAVIS DEPLOY KEY GENERATOR"
echo ""
echo "What it does : generate a new private and public key, add the public key as a deploy key
with write access to the origin remote github repository, encrypt the private key as
github_deploy_key.enc and add the configuration necessary to use it in .travis.yml file"
echo ""
echo "Use it like this:"
echo ""
echo " GITHUB_TOKEN=\`cat ~/secret/GITHUB_TOKEN\` ./generate_travis_deploy_key"
echo ""
echo "where ~/secret/GITHUB_TOKEN is a file containing a github token with write access to the current repository : (origin)"
echo ""
echo "You must have the travis executable installed on your system and available in the PATH"
echo ""
exit
fi
url="$(git config --get remote.origin.url)"
reponame="$(echo $url | cut -d/ -f2 | cut -d. -f1)"
# generate a new private and public key
ssh-keygen -t rsa -b 4096 -f github_deploy_key -N '' -C $url -q 1>/dev/null
pubkey="$(cat github_deploy_key.pub)"
# add the PUBLIC key to the github repository as a deploy key with write access
curl https://api.github.com/repos/konnectors/$reponame/keys -H "Authorization: token $GITHUB_TOKEN" --data @- << EOF
{
"title": "travis deploy key",
"key": "$pubkey",
"read_only": false
}
EOF
# use travis to encrypt the private key as github_deploy_key.enc and remove the private key
travis encrypt-file github_deploy_key --add --no-interactive -w /tmp/github_deploy_key -f
git add github_deploy_key.enc
# cleaning
rm github_deploy_key
rm github_deploy_key.pub
{
"version": "1.0.0",
"name": "Connector template",
"type": "konnector",
"language": "node",
"icon": "icon.png",
"slug": "template",
"source": "git://github.com/konnectors/cozy-konnector-template.git",
"editor": "Cozy",
"vendor_link": "Link to the target website",
"categories": ["other"],
"fields": {
"login": {
"type": "text"
},
"password": {
"type": "password"
},
"advancedFields": {
"folderPath": {
"advanced": true,
"isRequired": false
}
}
},
"data_types": [
"bill"
],
"screenshots": [],
"permissions": {
"bank operations": {
"type": "io.cozy.bank.operations"
},
"bills": {
"type": "io.cozy.bills"
},
"files": {
"type": "io.cozy.files"
},
"accounts": {
"type": "io.cozy.accounts",
"verbs": ["GET"]
}
},
"developer": {
"name": "Cozy Cloud",
"url": "https://cozy.io"
},
"langs": ["fr", "en"],
"locales": {
"fr": {
"short_description": "Template de connecteur",
"long_description": "Ce template récupère une liste de livres sur un site de scraping",
"permissions": {
"bank operations": {
"description": "Utilisé pour relier les factures à des operations bancaires"
},
"bills": {
"description": "Utilisé pour sauver les données des factures"
},
"files": {
"description": "Utilisé pour sauvegarder les factures"
},
"accounts": {
"description": "Utilisé pour obtenir les données du compte"
}
}
},
"en": {
"short_description": "Connector template",
"long_description": "This template fetches a list of books from a scraping website",
"permissions": {
"bank operations": {
"description": "Required to link bank operations to bills"
},
"bills": {
"description": "Required to save the bills data"
},
"files": {
"description": "Required to save the bills"
},
"accounts": {
"description": "Required to get the account's data"
}
}
}
},
"manifest_version": "2"
}
{
"name": "cozy-konnector-template",
"version": "1.0.0",
"description": "",
"repository": {
"type": "git",
"url": "git+https://github.com/konnectors/cozy-konnector-template.git"
},
"keywords": [],
"author": "Cozy Cloud",
"license": "AGPL-3.0",
"main": "./src/index.js",
"eslintConfig": {
"extends": [
"eslint-config-cozy-app"
]
},
"eslintIgnore": [
"build"
],
"scripts": {
"start": "node ./src/index.js",
"dev": "cozy-run-dev",
"standalone": "cozy-run-standalone",
"pretest": "npm run clean",
"test": "konitor testit .",
"check": "konitor check .",
"clean": "rm -rf ./data",
"build": "webpack",
"precommit": "yarn lint",
"lint": "eslint --fix .",
"deploy": "git-directory-deploy --directory build/ --branch ${DEPLOY_BRANCH:-build} --repo=${DEPLOY_REPOSITORY:-https://$GITHUB_TOKEN@github.com/konnectors/cozy-konnector-template.git}",
"cozyPublish": "cozy-app-publish --token $REGISTRY_TOKEN --build-commit $(git rev-parse ${DEPLOY_BRANCH:-build})",
"travisDeployKey": "./bin/generate_travis_deploy_key"
},
"dependencies": {
"cozy-konnector-libs": "4.10.2"
},
"devDependencies": {
"copy-webpack-plugin": "4.5.2",
"cozy-app-publish": "0.4.1-beta.3",
"cozy-jobs-cli": "1.4.9",
"eslint": "5.2.0",
"eslint-config-cozy-app": "0.10.0",
"git-directory-deploy": "1.5.1",
"husky": "0.14.3",
"konitor": "0.9.0",
"svgo": "1.0.5",
"webpack": "4.16.3",
"webpack-cli": "3.1.0"
}
}
{
"extends": ["cozy-konnector"]
}
const {
BaseKonnector,
requestFactory,
signin,
scrape,
saveBills,
log
} = require('cozy-konnector-libs')
const request = requestFactory({
debug: false,
cheerio: true,
json: false,
jar: true
})
const baseUrl = 'https://net-entreprise.fr/'
const serviceUrl = 'https://portail.net-entreprises.fr/priv/'
const urssafUrl = 'https://www.ti.urssaf.fr/'
module.exports = new BaseKonnector(start)
async function start(fields) {
let accData = this.getAccountData();
log('info', 'Authenticating ...');
//Auth and retrieve declaration URL
const urlDeclaration = await authenticate(fields.siret, fields.lastname, fields.firstname, fields.password);
log('info', 'Successfully logged in');
log('info', 'Get Declarations Parameters');
let params = await getDeclarationsParameters(urlDeclaration);
log('info', 'Get Declaration ');
const declarationList = await buildDeclarationList(params);
await getAllDeclaration(params,declarationList,accData);
}
async function authenticate(siret, lastname, firstname, password) {
const response = await signin({
url: `${baseUrl}`,
formSelector: 'form#form__connect',
formData: { j_siret:siret, j_nom:lastname, j_prenom:firstname, j_password:password },
// the validate function will check if
validate: (statusCode, body, fullResponse) => {
if(fullResponse.request['uri']['pathname'].startsWith("/auth/erreur"))
return false
else if(fullResponse.request['uri']['pathname'].startsWith("/priv/declarations")){
return true;
} else {
return false;
}
},
json : false,
headers: {
Referer: 'https://www.net-entreprises.fr/',
"Upgrade-Insecure-Requests": "1"
}
});
//Extract url for access to declarations
const doc = scrape(response('#service-10'), {
url: {
attr: 'href',
parse: href => `${serviceUrl}${href}`
}
});
return doc.url;
}
async function getDeclarationsParameters(url) {
let response = await request(url);
let doc = scrape(response('[name="form"]'),{
url: {
attr: 'action'
},
name: {
sel: 'input',
attr: 'name'
},
value: {
sel: 'input',
attr: 'value'
}
});
response = await request({
method: 'POST',
uri: doc.url,
form: { [doc.name] : doc.value }
});
doc = response('[name="menuform"]').serializeArray();
let form = {};
for(i = 0; i < doc.length; i++) {
form[[doc[i].name]] = doc[i].value;
}
return form;
}
async function buildDeclarationList(params) {
params.codepaye = 10;
params.echeance = 44;
params.listexi = '';
params.habpai = 'N';
params.habdev = 'N';
let completeList = [];
let partialList = await getList(params,'/action.encours_netmicro','href',1);
for(let i = 1; i < partialList.length;i++)
completeList.push(parseInt(partialList[i].period));
partialList = await getList(params,'/action.histo_netmicro','onclick',0);
for(let i = 0; i < partialList.length;i++)
completeList.push(parseInt(partialList[i].period));
return completeList;
}
async function getList(params,urlPart,subItem, splitPart) {
const data = await request({
method: 'POST',
uri: `${urssafUrl}/${urlPart}`,
form: params
});
let doc = scrape(data, {
period: {
sel: 'a',
attr: subItem,
parse: href => href.split("('")[1].split("','")[splitPart]
}
},'.menu_microsocial .subitem_menu_microsocial');
return doc;
}
async function getAllDeclaration(params,declarationList, accData) {
let exist = Object.keys(accData).length > 0;
if(!exist || accData.lastPeriod !== params.periode-1) {
let lastPeriod = declarationList.length;
if(exist) {
lastPeriod = declarationList.indexOf(accData.lastPeriod);
if(lastPeriod === -1)
lastPeriod = declarationList.length;
}
for(let i = 0; i < lastPeriod; i++) {
await getDeclaration(params,declarationList[i]);
}
}
}
async function getDeclaration(params, periode) {
console.log(periode);
params.periode = periode;
params.codepaye = 10;
params.echeance = 44;
params.listexi = '';
params.habpai = 'N';
params.habdev = 'N';
const data = await request({
method: 'POST',
uri: `${urssafUrl}/action.histo_netmicro`,
form: params
});
}
var path = require('path')
const CopyPlugin = require('copy-webpack-plugin')
const fs = require('fs')
const SvgoInstance = require('svgo')
const entry = require('./package.json').main
const svgo = new SvgoInstance()
let iconName
try {
iconName = JSON.parse(fs.readFileSync('manifest.konnector', 'utf8')).icon
// we run optimize only on SVG
if (!iconName.match(/\.svg$/)) iconName = null
} catch (e) {
// console.error(`Unable to read the icon path from manifest: ${e}`)
}
const appIconRX = iconName && new RegExp(`[^/]*/${iconName}`)
module.exports = {
entry,
target: 'node',
mode: 'none',
output: {
path: path.join(__dirname, 'build'),
filename: 'index.js'
},
plugins: [
new CopyPlugin([
{ from: 'manifest.konnector' },
{ from: 'package.json' },
{ from: 'README.md' },
{ from: 'assets', transform: optimizeSVGIcon },
{ from: '.travis.yml' },
{ from: 'LICENSE' }
])
]
}
function optimizeSVGIcon(buffer, path) {
if (appIconRX && path.match(appIconRX)) {
return svgo.optimize(buffer).then(resp => resp.data)
} else {
return buffer
}
}
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment